[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:05:55 dist-test-slave-chsk krb5kdc[4812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:05:55 dist-test-slave-chsk krb5kdc[4812](info): set up 2 sockets
Aug 08 21:05:55 dist-test-slave-chsk krb5kdc[4812](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 21:05:56.687391 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:05:56 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:05:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:05:56.814803 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35411
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:56.902307 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:56.902519 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:56.902542 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:56.903731 4826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:05:56.903772 4826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:56.903787 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:56.903817 4826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:05:56.903829 4826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:05:56.905412 4826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:56.905753 4826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:56.905968 4826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:56.912979 4832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:56.913002 4835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:56.913091 4831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:56.913744 4826 server_base.cc:1034] running on GCE node
I20260808 21:05:56.914064 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:56.914494 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:56.915642 4826 hybrid_clock.cc:648] HybridClock initialized: now 1786223156915624 us; error 73 us; skew 500 ppm
Aug 08 21:05:56 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223156, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:56.922026 4826 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:05:56.922719 4826 webserver.cc:460] Webserver started at http://127.4.177.126:33539/ using document root <none> and password file <none>
I20260808 21:05:56.922927 4826 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:56.922969 4826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:56.923072 4826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:56.924219 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "9a8449aa91d84408bd282c3a9e341670"
format_stamp: "Formatted at 2026-08-08 21:05:56 on dist-test-slave-chsk"
server_key: "e628da89b8172f33a9116834b0b34f88"
server_key_iv: "b4a4f7217f69209b69c1255065dd1859"
server_key_version: "encryptionkey@0"
I20260808 21:05:56.924525 4826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "9a8449aa91d84408bd282c3a9e341670"
format_stamp: "Formatted at 2026-08-08 21:05:56 on dist-test-slave-chsk"
server_key: "e628da89b8172f33a9116834b0b34f88"
server_key_iv: "b4a4f7217f69209b69c1255065dd1859"
server_key_version: "encryptionkey@0"
I20260808 21:05:56.925985 4826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:05:56.926697 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:56.926899 4826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:05:56.927008 4826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "9a8449aa91d84408bd282c3a9e341670"
format_stamp: "Formatted at 2026-08-08 21:05:56 on dist-test-slave-chsk"
server_key: "e628da89b8172f33a9116834b0b34f88"
server_key_iv: "b4a4f7217f69209b69c1255065dd1859"
server_key_version: "encryptionkey@0"
I20260808 21:05:56.927083 4826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:56.938369 4826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:56.940798 4826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:56.941048 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:56.946286 4826 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:35411
I20260808 21:05:56.946364 4896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:35411 every 8 connection(s)
I20260808 21:05:56.947193 4826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:05:56.948359 4897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:05:56.950603 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 4826
I20260808 21:05:56.950747 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:05:56.951748 4805 external_mini_cluster.cc:1428] Setting key cc02f0a3923d0519833b421e9a9965a2
I20260808 21:05:56.953382 4897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670: Bootstrap starting.
I20260808 21:05:56.955123 4897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670: Neither blocks nor log segments found. Creating new log.
I20260808 21:05:56.956028 4897 log.cc:824] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670: Log is configured to *not* fsync() on all Append() calls
I20260808 21:05:56.957123 4897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670: No bootstrap required, opened a new log
I20260808 21:05:56.961606 4897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } }
I20260808 21:05:56.961803 4897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:05:56.961880 4897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8449aa91d84408bd282c3a9e341670, State: Initialized, Role: FOLLOWER
I20260808 21:05:56.962066 4897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } }
I20260808 21:05:56.962256 4897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:05:56.962294 4897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:05:56.962334 4897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:05:56.963374 4897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } }
I20260808 21:05:56.963600 4897 leader_election.cc:302] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a8449aa91d84408bd282c3a9e341670; no voters:
I20260808 21:05:56.963832 4897 leader_election.cc:288] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:05:56.963892 4904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:05:56.964094 4904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [term 1 LEADER]: Becoming Leader. State: Replica: 9a8449aa91d84408bd282c3a9e341670, State: Running, Role: LEADER
I20260808 21:05:56.964243 4904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } }
I20260808 21:05:56.964391 4897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:05:56.964795 4904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a8449aa91d84408bd282c3a9e341670. Latest consensus state: current_term: 1 leader_uuid: "9a8449aa91d84408bd282c3a9e341670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } } }
I20260808 21:05:56.964879 4904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [sys.catalog]: This master's current role is: LEADER
I20260808 21:05:56.965015 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a8449aa91d84408bd282c3a9e341670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a8449aa91d84408bd282c3a9e341670" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35411 } } }
I20260808 21:05:56.965178 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670 [sys.catalog]: This master's current role is: LEADER
I20260808 21:05:56.966871 4908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 08 21:05:56 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:56.968152 4908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:05:56.970621 4908 catalog_manager.cc:1347] Generated new cluster ID: db84e948a278447580fb436dc0842eb8
I20260808 21:05:56.970676 4908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:05:56.974432 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:56.953398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.4.177.126:35411)
0808 21:05:56.953743 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:56.953753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:56.953957 (+ 204us) server_negotiation.cc:407] Connection header received
0808 21:05:56.956899 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:56.956906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:56.957033 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:56.957208 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:56.959092 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:56.960901 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:56.962158 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:56.962882 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:56.963067 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:56.968657 (+ 5590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:56.968661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:56.968662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:56.968683 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:56.972062 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:56.972578 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:56.972587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:56.972588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:56.972714 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:56.973016 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:56.973017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:56.973018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:56.973443 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:56.973632 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:56.973934 (+ 302us) server_negotiation.cc:299] Negotiation successful
0808 21:05:56.974119 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260808 21:05:56.993307 4908 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:05:56.993976 4908 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:05:56.997731 4908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a8449aa91d84408bd282c3a9e341670: Generated new TSK 0
I20260808 21:05:56.997927 4908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:05:57.030782 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:57.130530 4925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:57.130755 4925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:57.130781 4925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:57.132078 4925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:57.132120 4925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:57.132153 4925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:05:57.133850 4925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:57.134159 4925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:57.134416 4925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:57.140378 4930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.140451 4931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.140769 4933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:57.141338 4925 server_base.cc:1034] running on GCE node
I20260808 21:05:57.141566 4925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:57.141813 4925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:57.142978 4925 hybrid_clock.cc:648] HybridClock initialized: now 1786223157142965 us; error 28 us; skew 500 ppm
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:57.146078 4925 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:05:57.146539 4925 webserver.cc:460] Webserver started at http://127.4.177.65:42443/ using document root <none> and password file <none>
I20260808 21:05:57.146793 4925 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:57.146863 4925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:57.147012 4925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:57.148257 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "6eaaf0f92d314e94819a7c6ab9d7dc1c"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "0da8812f54c5adc27d9c4b4386d98afe"
server_key_iv: "4119b6966d0a8fdfe54bb9a7e8eece33"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.148643 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "6eaaf0f92d314e94819a7c6ab9d7dc1c"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "0da8812f54c5adc27d9c4b4386d98afe"
server_key_iv: "4119b6966d0a8fdfe54bb9a7e8eece33"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.150198 4925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:05:57.151104 4940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.151326 4925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:05:57.151427 4925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "6eaaf0f92d314e94819a7c6ab9d7dc1c"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "0da8812f54c5adc27d9c4b4386d98afe"
server_key_iv: "4119b6966d0a8fdfe54bb9a7e8eece33"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.151515 4925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:57.162824 4925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:57.165102 4925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:57.165325 4925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:57.165671 4925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:57.166085 4925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:57.166169 4925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.166213 4925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:57.166247 4925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.179481 4925 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:45265
I20260808 21:05:57.179661 5062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:45265 every 8 connection(s)
I20260808 21:05:57.179943 4925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:57.187325 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 4925
I20260808 21:05:57.187427 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:05:57.187577 4805 external_mini_cluster.cc:1428] Setting key 2782ab057eef87e857b66169acf3a0d4
I20260808 21:05:57.190091 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.180928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:36241 (local address 127.4.177.126:35411)
0808 21:05:57.181000 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:57.181002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:57.181260 (+ 258us) server_negotiation.cc:407] Connection header received
0808 21:05:57.181852 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:57.181854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:57.181866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:57.181928 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:57.183023 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.183547 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.184494 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.184698 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.184748 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.186829 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:57.186831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:57.186831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:57.186842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:57.188604 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.189116 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.189117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.189117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.189168 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.189458 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.189459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.189459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.189566 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:57.189623 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:57.189948 (+ 325us) server_negotiation.cc:299] Negotiation successful
0808 21:05:57.190025 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:05:57.190521 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.180901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35411 (local address 127.4.177.65:36241)
0808 21:05:57.181115 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:57.181162 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:57.181743 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:57.181980 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:57.181983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:57.182736 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:57.182932 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.182937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.183625 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.183626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.184416 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.184418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.184769 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.184769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.184869 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.185340 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:57.185348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:57.186719 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:57.188699 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.188707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.188710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.189032 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.189240 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.189241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.189241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.189393 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.189638 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:57.189644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:57.189812 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 21:05:57.189929 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 21:05:57.190110 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
I20260808 21:05:57.191203 5063 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35411
I20260808 21:05:57.191318 5063 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:57.191500 5063 heartbeater.cc:511] Master 127.4.177.126:35411 requested a full tablet report, sending...
I20260808 21:05:57.192469 4860 ts_manager.cc:194] Registered new tserver with Master: 6eaaf0f92d314e94819a7c6ab9d7dc1c (127.4.177.65:45265)
I20260808 21:05:57.193668 4860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:36241
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:05:57.215977 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:57.304059 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:57.304316 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:57.304342 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:57.305495 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:57.305533 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:57.305559 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:05:57.306960 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:57.307216 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:57.307422 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:05:57.312844 5068 server_base.cc:1034] running on GCE node
W20260808 21:05:57.312827 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.312819 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.312911 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:57.313271 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:57.313539 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:57.314687 5068 hybrid_clock.cc:648] HybridClock initialized: now 1786223157314674 us; error 31 us; skew 500 ppm
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:57.317354 5068 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:05:57.317816 5068 webserver.cc:460] Webserver started at http://127.4.177.66:35817/ using document root <none> and password file <none>
I20260808 21:05:57.318051 5068 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:57.318123 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:57.318277 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:57.319504 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "b12ee0f4ce644c6a9fec4976c562a1e9"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "6d030df002a51b587704ab1c277a383c"
server_key_iv: "55f9cb7a685b1cee9227cf87dd6b3a79"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.319881 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "b12ee0f4ce644c6a9fec4976c562a1e9"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "6d030df002a51b587704ab1c277a383c"
server_key_iv: "55f9cb7a685b1cee9227cf87dd6b3a79"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.321338 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:05:57.322299 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.322497 5068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:05:57.322620 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "b12ee0f4ce644c6a9fec4976c562a1e9"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "6d030df002a51b587704ab1c277a383c"
server_key_iv: "55f9cb7a685b1cee9227cf87dd6b3a79"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.322702 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:57.341562 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:57.343722 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:57.343891 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:57.344158 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:57.344482 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:57.344539 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.344594 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:57.344625 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.352797 5068 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:44063
I20260808 21:05:57.352919 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:44063 every 8 connection(s)
I20260808 21:05:57.353220 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:57.361950 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5068
I20260808 21:05:57.362082 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:05:57.362272 4805 external_mini_cluster.cc:1428] Setting key 472927da288f31725d2e81360d501216
I20260808 21:05:57.362349 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.354120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:50923 (local address 127.4.177.126:35411)
0808 21:05:57.354190 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:57.354192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:57.354439 (+ 247us) server_negotiation.cc:407] Connection header received
0808 21:05:57.354991 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:57.354992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:57.355002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:57.355054 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:57.356060 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.356556 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.357482 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.357665 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.357716 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.359775 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:57.359776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:57.359777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:57.359782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:57.360992 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.361352 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.361353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.361353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.361400 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.361611 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.361612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.361612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.361723 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:57.361823 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:57.362114 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 21:05:57.362228 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 21:05:57.362515 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.354085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35411 (local address 127.4.177.66:50923)
0808 21:05:57.354313 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:57.354350 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:57.354897 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:57.355107 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:57.355110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:57.355786 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:57.355981 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.355986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.356637 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.356637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.357407 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.357409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.357737 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.357737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.357832 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.358283 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:57.358291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:57.359695 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:57.361060 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.361065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.361067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.361268 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.361456 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.361457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.361457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.361549 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.361812 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:57.361817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:57.361925 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 21:05:57.362062 (+ 137us) client_negotiation.cc:240] Negotiation successful
0808 21:05:57.362232 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":151,"threads_started":1}
I20260808 21:05:57.363178 5206 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35411
I20260808 21:05:57.363269 5206 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:57.363461 5206 heartbeater.cc:511] Master 127.4.177.126:35411 requested a full tablet report, sending...
I20260808 21:05:57.363844 4860 ts_manager.cc:194] Registered new tserver with Master: b12ee0f4ce644c6a9fec4976c562a1e9 (127.4.177.66:44063)
I20260808 21:05:57.364250 4860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:50923
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:05:57.389722 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:57.476419 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:57.476621 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:57.476646 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:57.477771 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:57.477813 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:57.477867 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:05:57.479377 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:57.479610 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:57.479837 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:57.484710 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.484724 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:57.484715 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:57.485030 5212 server_base.cc:1034] running on GCE node
I20260808 21:05:57.485220 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:57.485526 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:57.486671 5212 hybrid_clock.cc:648] HybridClock initialized: now 1786223157486655 us; error 42 us; skew 500 ppm
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:57.489691 5212 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:05:57.490195 5212 webserver.cc:460] Webserver started at http://127.4.177.67:34481/ using document root <none> and password file <none>
I20260808 21:05:57.490375 5212 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:57.490417 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:57.490504 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:57.491570 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "d27cc6647a8d47eda3aefee27343bef3"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "70905e3a418e3d8321bd3724757644cd"
server_key_iv: "446a51a02714e2e0fbe7d24bb9d18193"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.491891 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "d27cc6647a8d47eda3aefee27343bef3"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "70905e3a418e3d8321bd3724757644cd"
server_key_iv: "446a51a02714e2e0fbe7d24bb9d18193"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.493304 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:05:57.493955 5227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.494167 5212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:05:57.494259 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "d27cc6647a8d47eda3aefee27343bef3"
format_stamp: "Formatted at 2026-08-08 21:05:57 on dist-test-slave-chsk"
server_key: "70905e3a418e3d8321bd3724757644cd"
server_key_iv: "446a51a02714e2e0fbe7d24bb9d18193"
server_key_version: "encryptionkey@0"
I20260808 21:05:57.494313 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:57.509485 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:57.511482 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:57.511596 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:57.511792 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:57.512118 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:57.512153 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.512199 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:57.512226 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:57.520249 5212 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:45625
I20260808 21:05:57.520309 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:45625 every 8 connection(s)
I20260808 21:05:57.520627 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:05:57.525755 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5212
I20260808 21:05:57.525846 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:05:57.525997 4805 external_mini_cluster.cc:1428] Setting key 5aba74106ba417a90b971d0e5f5c6ee7
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:57.530380 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.521410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:55547 (local address 127.4.177.126:35411)
0808 21:05:57.521498 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:57.521501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:57.521706 (+ 205us) server_negotiation.cc:407] Connection header received
0808 21:05:57.522233 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:57.522234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:57.522244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:57.522297 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:57.523280 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.523781 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.524715 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.524895 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.524945 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.527502 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:57.527504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:57.527504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:57.527512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:57.528953 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.529370 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.529371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.529372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.529434 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.529707 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.529708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.529708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.529824 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:57.529900 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:57.530180 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 21:05:57.530263 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 21:05:57.530537 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.521389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35411 (local address 127.4.177.67:55547)
0808 21:05:57.521584 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:57.521621 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:57.522148 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:57.522349 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:57.522352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:57.522923 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:57.523129 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.523133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.523859 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.523859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.524644 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:57.524646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.524966 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.524967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:57.525062 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.525517 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:57.525526 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:57.527398 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:57.529058 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.529065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.529067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.529290 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.529513 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:57.529514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:57.529514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:57.529632 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:57.529890 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:57.529897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:57.530035 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 21:05:57.530147 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 21:05:57.530301 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260808 21:05:57.531013 5350 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35411
I20260808 21:05:57.531106 5350 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:57.531324 5350 heartbeater.cc:511] Master 127.4.177.126:35411 requested a full tablet report, sending...
I20260808 21:05:57.531697 4860 ts_manager.cc:194] Registered new tserver with Master: d27cc6647a8d47eda3aefee27343bef3 (127.4.177.67:45625)
I20260808 21:05:57.532035 4860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:55547
I20260808 21:05:57.538708 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:05:57.563897 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.556256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.4.177.65:45265)
0808 21:05:57.556457 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:57.556462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:57.556472 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:05:57.556497 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:57.556499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:57.556513 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:57.556605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:57.557140 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.557829 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.558720 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.559012 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.559066 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.561205 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:57.561212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:57.561213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:57.561219 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:57.562621 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.562927 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.562929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.562929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.562982 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.563354 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.563354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.563355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.563473 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:57.563572 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:57.563662 (+ 90us) server_negotiation.cc:299] Negotiation successful
0808 21:05:57.563734 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":122,"threads_started":1}
W20260808 21:05:57.564379 4970 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38948
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[4812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:05:57.586720 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:57.580377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.4.177.65:45265)
0808 21:05:57.580458 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:57.580459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:57.580538 (+ 79us) server_negotiation.cc:407] Connection header received
0808 21:05:57.580581 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:57.580582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:57.580594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:57.580651 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:57.581169 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.581622 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.582313 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:57.582497 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:57.582547 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:57.584663 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:57.584664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:57.584664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:57.584669 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:57.585842 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.586136 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.586137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.586137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.586183 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:57.586342 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:57.586342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:57.586342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:57.586422 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:57.586509 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:57.586573 (+ 64us) server_negotiation.cc:299] Negotiation successful
0808 21:05:57.586631 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 21:05:57.587563 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 4925
I20260808 21:05:57.594034 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5068
I20260808 21:05:57.600560 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5212
I20260808 21:05:57.605873 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 4826
2026-08-08T21:05:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2051 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[5365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[5365](info): set up 2 sockets
Aug 08 21:05:57 dist-test-slave-chsk krb5kdc[5365](info): commencing operation
krb5kdc: starting...
W20260808 21:05:58.679713 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:05:58 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:05:58.807785 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38283
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:58.891870 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:58.892156 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:58.892207 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:58.893440 5379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:05:58.893519 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:58.893543 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:58.893564 5379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:05:58.893594 5379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:05:58.895208 5379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:58.895516 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:58.895764 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:58.900913 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:58.900915 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:58.900915 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:58.901543 5379 server_base.cc:1034] running on GCE node
I20260808 21:05:58.901798 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:58.902070 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:58.903208 5379 hybrid_clock.cc:648] HybridClock initialized: now 1786223158903194 us; error 32 us; skew 500 ppm
Aug 08 21:05:58 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223158, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:58.905828 5379 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:05:58.906329 5379 webserver.cc:460] Webserver started at http://127.4.177.126:36803/ using document root <none> and password file <none>
I20260808 21:05:58.906575 5379 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:58.906625 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:58.906762 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:58.908059 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "ec0098e6f5b24fbd89ef06d144d25f91"
format_stamp: "Formatted at 2026-08-08 21:05:58 on dist-test-slave-chsk"
server_key: "15565b4cd34e2e20a68a735b4e638d16"
server_key_iv: "b9e9042fb6ca88e708183ea089a0887d"
server_key_version: "encryptionkey@0"
I20260808 21:05:58.908488 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "ec0098e6f5b24fbd89ef06d144d25f91"
format_stamp: "Formatted at 2026-08-08 21:05:58 on dist-test-slave-chsk"
server_key: "15565b4cd34e2e20a68a735b4e638d16"
server_key_iv: "b9e9042fb6ca88e708183ea089a0887d"
server_key_version: "encryptionkey@0"
I20260808 21:05:58.909952 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:05:58.910864 5396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:58.911082 5379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:05:58.911226 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "ec0098e6f5b24fbd89ef06d144d25f91"
format_stamp: "Formatted at 2026-08-08 21:05:58 on dist-test-slave-chsk"
server_key: "15565b4cd34e2e20a68a735b4e638d16"
server_key_iv: "b9e9042fb6ca88e708183ea089a0887d"
server_key_version: "encryptionkey@0"
I20260808 21:05:58.911293 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:58.931550 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:58.934094 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:58.934231 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:58.938697 5379 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:38283
I20260808 21:05:58.938781 5451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:38283 every 8 connection(s)
I20260808 21:05:58.939170 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:05:58.940158 5452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:05:58.942667 5452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91: Bootstrap starting.
I20260808 21:05:58.943205 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5379
I20260808 21:05:58.943276 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:05:58.943426 5452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91: Neither blocks nor log segments found. Creating new log.
I20260808 21:05:58.943445 4805 external_mini_cluster.cc:1428] Setting key 3f7c7166f964040a8ca059716449a73c
I20260808 21:05:58.943737 5452 log.cc:824] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91: Log is configured to *not* fsync() on all Append() calls
I20260808 21:05:58.944521 5452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91: No bootstrap required, opened a new log
I20260808 21:05:58.946878 5452 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } }
I20260808 21:05:58.947063 5452 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:05:58.947101 5452 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0098e6f5b24fbd89ef06d144d25f91, State: Initialized, Role: FOLLOWER
I20260808 21:05:58.947252 5452 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } }
I20260808 21:05:58.947338 5452 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:05:58.947367 5452 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:05:58.947412 5452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:05:58.948066 5452 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } }
I20260808 21:05:58.948154 5452 leader_election.cc:302] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0098e6f5b24fbd89ef06d144d25f91; no voters:
I20260808 21:05:58.948359 5452 leader_election.cc:288] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:05:58.948441 5457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:05:58.948652 5452 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:05:58.948652 5457 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [term 1 LEADER]: Becoming Leader. State: Replica: ec0098e6f5b24fbd89ef06d144d25f91, State: Running, Role: LEADER
I20260808 21:05:58.948866 5457 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } }
I20260808 21:05:58.949255 5458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } } }
I20260808 21:05:58.949348 5458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [sys.catalog]: This master's current role is: LEADER
I20260808 21:05:58.949573 5468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:05:58.950095 5468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:05:58.950773 5459 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0098e6f5b24fbd89ef06d144d25f91. Latest consensus state: current_term: 1 leader_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0098e6f5b24fbd89ef06d144d25f91" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38283 } } }
I20260808 21:05:58.950886 5459 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91 [sys.catalog]: This master's current role is: LEADER
Aug 08 21:05:58 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:58.952131 5468 catalog_manager.cc:1347] Generated new cluster ID: 8cef15cee8ec4c01bd4e71c4d70e8415
I20260808 21:05:58.952229 5468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:05:58.955026 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:58.944032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.4.177.126:38283)
0808 21:05:58.944236 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:58.944247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:58.944284 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:05:58.944804 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:58.944811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:58.944860 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:58.944984 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:58.945806 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:58.946697 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:58.947785 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:58.948277 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:58.948389 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:58.951675 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:58.951679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:58.951679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:58.951691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:58.953470 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:58.953859 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:58.953862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:58.953862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:58.953956 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:58.954176 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:58.954177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:58.954177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:58.954412 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:58.954589 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:58.954676 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 21:05:58.954831 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260808 21:05:58.998198 5468 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:05:58.998821 5468 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:05:59.009035 5468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0098e6f5b24fbd89ef06d144d25f91: Generated new TSK 0
I20260808 21:05:59.009364 5468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:05:59.054550 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:59.161756 5480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:59.161967 5480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:59.161991 5480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:59.163156 5480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:59.163194 5480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:59.163221 5480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:05:59.164762 5480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:59.165019 5480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:59.165287 5480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:59.170615 5489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:59.170639 5485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:59.170635 5487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:59.170902 5480 server_base.cc:1034] running on GCE node
I20260808 21:05:59.171213 5480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:59.171437 5480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:59.172575 5480 hybrid_clock.cc:648] HybridClock initialized: now 1786223159172558 us; error 33 us; skew 500 ppm
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:59.175315 5480 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:05:59.175863 5480 webserver.cc:460] Webserver started at http://127.4.177.65:40457/ using document root <none> and password file <none>
I20260808 21:05:59.176126 5480 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:59.176195 5480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:59.176345 5480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:59.177400 5480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "913164c1924b45b49858468f12bc9510"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "3395a4b981e28ca9eab5d901519a43e7"
server_key_iv: "9bb35f8036a2e361f47f436018458c51"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.177765 5480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "913164c1924b45b49858468f12bc9510"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "3395a4b981e28ca9eab5d901519a43e7"
server_key_iv: "9bb35f8036a2e361f47f436018458c51"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.179210 5480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:05:59.180145 5497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.180380 5480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:05:59.180497 5480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "913164c1924b45b49858468f12bc9510"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "3395a4b981e28ca9eab5d901519a43e7"
server_key_iv: "9bb35f8036a2e361f47f436018458c51"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.180596 5480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:59.190860 5480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:59.193099 5480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:59.193291 5480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:59.193562 5480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:59.193902 5480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:59.193956 5480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.194003 5480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:59.194026 5480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.200508 5480 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:41123
I20260808 21:05:59.200603 5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:41123 every 8 connection(s)
I20260808 21:05:59.200968 5480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:05:59.201668 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5480
I20260808 21:05:59.201782 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:05:59.201975 4805 external_mini_cluster.cc:1428] Setting key 19bf8e93abc8a683c09ff32b7bb069cd
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
I20260808 21:05:59.212764 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.201785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:41237 (local address 127.4.177.126:38283)
0808 21:05:59.201851 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.201854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.202276 (+ 422us) server_negotiation.cc:407] Connection header received
0808 21:05:59.203633 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.203634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.203649 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.203732 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.204410 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.205010 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.206044 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.206323 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.206388 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.209122 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.209123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.209124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.209134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.211024 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.211556 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.211558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.211559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.211625 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.211937 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.211938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.211939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.212110 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.212196 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.212495 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.212611 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:05:59.212817 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.201784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38283 (local address 127.4.177.65:41237)
0808 21:05:59.202098 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.202146 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.203517 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.203790 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.203793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.204072 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:59.204277 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.204282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.205100 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.205101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.205941 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.205943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.206431 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.206432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.206566 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.207236 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:59.207245 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:59.208997 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:59.211128 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.211136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.211191 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.211466 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.211724 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.211725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.211725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.211848 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.212189 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:59.212196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:59.212342 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.212427 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.212569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260808 21:05:59.213424 5622 heartbeater.cc:347] Connected to a master server at 127.4.177.126:38283
I20260808 21:05:59.213549 5622 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:59.213994 5622 heartbeater.cc:511] Master 127.4.177.126:38283 requested a full tablet report, sending...
I20260808 21:05:59.214516 5415 ts_manager.cc:194] Registered new tserver with Master: 913164c1924b45b49858468f12bc9510 (127.4.177.65:41123)
I20260808 21:05:59.215365 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:41237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:05:59.232102 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:59.317854 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:59.318087 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:59.318114 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:59.319360 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:59.319406 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:59.319433 5628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:05:59.320952 5628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:59.321161 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:59.321365 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:59.326515 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:59.326558 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:59.326555 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:59.326622 5628 server_base.cc:1034] running on GCE node
I20260808 21:05:59.326969 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:59.327348 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:59.328544 5628 hybrid_clock.cc:648] HybridClock initialized: now 1786223159328515 us; error 41 us; skew 500 ppm
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:59.331238 5628 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:05:59.331764 5628 webserver.cc:460] Webserver started at http://127.4.177.66:38609/ using document root <none> and password file <none>
I20260808 21:05:59.331998 5628 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:59.332072 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:59.332228 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:59.333297 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "2206866a221347aaa742f5b1e098fef7"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "32e570a63f0ad70a99d471f457881826"
server_key_iv: "7c90530a7abdf71549fd034b07c37576"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.333666 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "2206866a221347aaa742f5b1e098fef7"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "32e570a63f0ad70a99d471f457881826"
server_key_iv: "7c90530a7abdf71549fd034b07c37576"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.335036 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:05:59.335843 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.336053 5628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:05:59.336161 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "2206866a221347aaa742f5b1e098fef7"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "32e570a63f0ad70a99d471f457881826"
server_key_iv: "7c90530a7abdf71549fd034b07c37576"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.336258 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:59.353401 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:59.355388 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:59.355562 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:59.355800 5628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:59.356106 5628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:59.356163 5628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.356205 5628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:59.356231 5628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.361924 5628 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:44725
I20260808 21:05:59.361984 5764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:44725 every 8 connection(s)
I20260808 21:05:59.362344 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:59.368611 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5628
I20260808 21:05:59.368709 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:05:59.368870 4805 external_mini_cluster.cc:1428] Setting key 18cf5a8c1520fd20b3fe5bde7da2320c
I20260808 21:05:59.371924 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.363087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:39821 (local address 127.4.177.126:38283)
0808 21:05:59.363197 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.363200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.363363 (+ 163us) server_negotiation.cc:407] Connection header received
0808 21:05:59.364421 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.364422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.364432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.364486 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.365036 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.365535 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.366478 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.366660 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.366712 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.368899 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.368901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.368902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.368909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.370462 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.370901 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.370902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.370902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.370959 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.371274 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.371275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.371275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.371392 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.371470 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.371748 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.371836 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 21:05:59.372100 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.363033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38283 (local address 127.4.177.66:39821)
0808 21:05:59.363237 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.363281 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.364333 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.364535 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.364538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.364787 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:59.364971 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.364974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.365620 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.365620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.366403 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.366405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.366731 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.366732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.366824 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.367332 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:59.367340 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:59.368802 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:59.370566 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.370573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.370575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.370807 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.371035 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.371036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.371037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.371210 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.371460 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:59.371467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:59.371605 (+ 138us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.371712 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.371865 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260808 21:05:59.372546 5766 heartbeater.cc:347] Connected to a master server at 127.4.177.126:38283
I20260808 21:05:59.372658 5766 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:59.372833 5766 heartbeater.cc:511] Master 127.4.177.126:38283 requested a full tablet report, sending...
I20260808 21:05:59.373226 5415 ts_manager.cc:194] Registered new tserver with Master: 2206866a221347aaa742f5b1e098fef7 (127.4.177.66:44725)
I20260808 21:05:59.373667 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:39821
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:05:59.395579 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:05:59.478883 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:05:59.479139 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:05:59.479204 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:05:59.480409 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:05:59.480479 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:05:59.480527 5771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:05:59.482072 5771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:05:59.482357 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:05:59.482590 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:05:59.487614 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:05:59.487630 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:59.487748 5771 server_base.cc:1034] running on GCE node
W20260808 21:05:59.487614 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:05:59.488178 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:05:59.488430 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:05:59.489583 5771 hybrid_clock.cc:648] HybridClock initialized: now 1786223159489570 us; error 35 us; skew 500 ppm
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:05:59.492238 5771 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:05:59.492745 5771 webserver.cc:460] Webserver started at http://127.4.177.67:40441/ using document root <none> and password file <none>
I20260808 21:05:59.492986 5771 fs_manager.cc:359] Metadata directory not provided
I20260808 21:05:59.493033 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:05:59.493208 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:05:59.494252 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "4bc93ba09acf48bcbcbec4533a103439"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "d2ae11a680dd31dc39b15ba9b250f6e0"
server_key_iv: "2d133bc7762199088f7b935de51e364a"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.494681 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "4bc93ba09acf48bcbcbec4533a103439"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "d2ae11a680dd31dc39b15ba9b250f6e0"
server_key_iv: "2d133bc7762199088f7b935de51e364a"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.496120 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:05:59.496867 5788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.497045 5771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:05:59.497139 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "4bc93ba09acf48bcbcbec4533a103439"
format_stamp: "Formatted at 2026-08-08 21:05:59 on dist-test-slave-chsk"
server_key: "d2ae11a680dd31dc39b15ba9b250f6e0"
server_key_iv: "2d133bc7762199088f7b935de51e364a"
server_key_version: "encryptionkey@0"
I20260808 21:05:59.497215 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:05:59.509795 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:05:59.511916 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:05:59.512037 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:05:59.512223 5771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:05:59.512539 5771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:05:59.512575 5771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.512634 5771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:05:59.512661 5771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:05:59.519271 5771 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40493
I20260808 21:05:59.519307 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40493 every 8 connection(s)
I20260808 21:05:59.519675 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:05:59.521245 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5771
I20260808 21:05:59.521327 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:05:59.521482 4805 external_mini_cluster.cc:1428] Setting key f8843b8caaf71bf6139b7183987adcca
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:05:59.529572 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.520504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:56187 (local address 127.4.177.126:38283)
0808 21:05:59.520580 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.520582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.520810 (+ 228us) server_negotiation.cc:407] Connection header received
0808 21:05:59.521952 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.521953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.521967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.522035 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.522770 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.523365 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.524382 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.524582 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.524674 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.526727 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.526728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.526729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.526736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.528114 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.528586 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.528587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.528588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.528647 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.528919 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.528920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.528920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.529037 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.529114 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.529394 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.529477 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:05:59.529776 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.520499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38283 (local address 127.4.177.67:56187)
0808 21:05:59.520675 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.520717 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.521849 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.522111 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.522115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.522401 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:05:59.522651 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.522656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.523459 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.523460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.524277 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.524279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.524677 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.524678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.524813 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.525279 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:05:59.525287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:05:59.526628 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:05:59.528221 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.528227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.528229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.528511 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.528728 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:05:59.528729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:05:59.528729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:05:59.528844 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:05:59.529104 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:05:59.529111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:05:59.529260 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.529372 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.529534 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260808 21:05:59.530531 5906 heartbeater.cc:347] Connected to a master server at 127.4.177.126:38283
I20260808 21:05:59.530619 5906 heartbeater.cc:464] Registering TS with master...
I20260808 21:05:59.530807 5906 heartbeater.cc:511] Master 127.4.177.126:38283 requested a full tablet report, sending...
I20260808 21:05:59.531328 5415 ts_manager.cc:194] Registered new tserver with Master: 4bc93ba09acf48bcbcbec4533a103439 (127.4.177.67:40493)
I20260808 21:05:59.531734 5415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:56187
I20260808 21:05:59.533282 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:05:59.540891 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.535163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.4.177.126:38283)
0808 21:05:59.535218 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.535220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.535317 (+ 97us) server_negotiation.cc:407] Connection header received
0808 21:05:59.535356 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.535357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.535366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.535412 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.535919 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.536376 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.537056 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.537225 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.537271 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.538466 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.538467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.538467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.538473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.539844 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.540221 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.540222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.540222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.540273 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.540422 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.540422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.540422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.540552 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.540629 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.540729 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.540790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 21:05:59.543885 5415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:05:59.545634 5415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:05:59.553078 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.548618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41123 (local address 127.0.0.1:37910)
0808 21:05:59.548854 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.548908 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.548985 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.549486 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.549489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.549491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:05:59.549558 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.549560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.550927 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.550927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.552100 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.552102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.552783 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.552785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.552867 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.552881 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.552949 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.553000 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260808 21:05:59.553085 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.548764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.4.177.65:41123)
0808 21:05:59.549149 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.549153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.549165 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:05:59.549207 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.549209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.549233 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.549323 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:05:59.549679 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.550816 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.552204 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.552692 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.552784 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.552843 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.552963 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.553000 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":71,"threads_started":1}
I20260808 21:05:59.553366 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.548630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:44725 (local address 127.0.0.1:41562)
0808 21:05:59.548931 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.548943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.548997 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.549555 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.549556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.549559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:05:59.549618 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.549620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.550723 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.550724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.552099 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.552102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.553110 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.553111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.553172 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.553194 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.553251 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.553300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":33,"thread_start_us":118,"threads_started":1}
I20260808 21:05:59.553653 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.548730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.4.177.66:44725)
0808 21:05:59.549109 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.549113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.549132 (+ 19us) server_negotiation.cc:407] Connection header received
0808 21:05:59.549164 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.549166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.549177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.549438 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:05:59.549701 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.550610 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.552194 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.552964 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.553022 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.553069 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.553518 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.553560 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":216,"threads_started":1}
I20260808 21:05:59.553891 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.548877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40493 (local address 127.0.0.1:46236)
0808 21:05:59.549299 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.549308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.549539 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.550004 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.550005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.550007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:05:59.550069 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.550070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.551250 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.551251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.552117 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.552120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.553094 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.553095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.553560 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.553565 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.553632 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.553671 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":75,"threads_started":1}
I20260808 21:05:59.553958 5550 tablet_service.cc:1467] Processing CreateTablet for tablet bd32102374044d2a972e3646bb341164 (DEFAULT_TABLE table=test-table [id=3105d17d3ea54454869ef87cdc6f0edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:05:59.554340 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32102374044d2a972e3646bb341164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:05:59.554316 5696 tablet_service.cc:1467] Processing CreateTablet for tablet bd32102374044d2a972e3646bb341164 (DEFAULT_TABLE table=test-table [id=3105d17d3ea54454869ef87cdc6f0edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:05:59.554627 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32102374044d2a972e3646bb341164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:05:59.555256 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.549125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.4.177.67:40493)
0808 21:05:59.549410 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.549414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.549649 (+ 235us) server_negotiation.cc:407] Connection header received
0808 21:05:59.549698 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.549700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.549716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.549996 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:05:59.550154 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.551110 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.552226 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.552926 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.553001 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.553056 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.553652 (+ 596us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.553696 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260808 21:05:59.555780 5838 tablet_service.cc:1467] Processing CreateTablet for tablet bd32102374044d2a972e3646bb341164 (DEFAULT_TABLE table=test-table [id=3105d17d3ea54454869ef87cdc6f0edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:05:59.556056 5838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd32102374044d2a972e3646bb341164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:05:59.557966 5925 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Bootstrap starting.
I20260808 21:05:59.558856 5925 tablet_bootstrap.cc:654] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Neither blocks nor log segments found. Creating new log.
I20260808 21:05:59.559101 5927 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Bootstrap starting.
I20260808 21:05:59.559669 5927 tablet_bootstrap.cc:654] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Neither blocks nor log segments found. Creating new log.
I20260808 21:05:59.559774 5925 log.cc:824] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Log is configured to *not* fsync() on all Append() calls
I20260808 21:05:59.560173 5927 log.cc:824] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Log is configured to *not* fsync() on all Append() calls
I20260808 21:05:59.560555 5929 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Bootstrap starting.
I20260808 21:05:59.561141 5927 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: No bootstrap required, opened a new log
I20260808 21:05:59.561224 5927 ts_tablet_manager.cc:1397] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:05:59.561252 5929 tablet_bootstrap.cc:654] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Neither blocks nor log segments found. Creating new log.
I20260808 21:05:59.561455 5925 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: No bootstrap required, opened a new log
I20260808 21:05:59.561537 5925 ts_tablet_manager.cc:1397] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260808 21:05:59.561820 5929 log.cc:824] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:05:59.562675 5929 tablet_bootstrap.cc:492] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: No bootstrap required, opened a new log
I20260808 21:05:59.562747 5929 ts_tablet_manager.cc:1397] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:05:59.563063 5927 raft_consensus.cc:348] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.563294 5927 raft_consensus.cc:374] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:05:59.563344 5927 raft_consensus.cc:729] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc93ba09acf48bcbcbec4533a103439, State: Initialized, Role: FOLLOWER
I20260808 21:05:59.563458 5927 consensus_queue.cc:260] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.563594 5925 raft_consensus.cc:348] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.563738 5927 ts_tablet_manager.cc:1428] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:05:59.563776 5925 raft_consensus.cc:374] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:05:59.563824 5925 raft_consensus.cc:729] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913164c1924b45b49858468f12bc9510, State: Initialized, Role: FOLLOWER
I20260808 21:05:59.563928 5906 heartbeater.cc:503] Master 127.4.177.126:38283 was elected leader, sending a full tablet report...
I20260808 21:05:59.563925 5925 consensus_queue.cc:260] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.564244 5622 heartbeater.cc:503] Master 127.4.177.126:38283 was elected leader, sending a full tablet report...
I20260808 21:05:59.564276 5925 ts_tablet_manager.cc:1428] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:05:59.565205 5929 raft_consensus.cc:348] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.565330 5929 raft_consensus.cc:374] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:05:59.565393 5929 raft_consensus.cc:729] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2206866a221347aaa742f5b1e098fef7, State: Initialized, Role: FOLLOWER
I20260808 21:05:59.565529 5929 consensus_queue.cc:260] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.565793 5929 ts_tablet_manager.cc:1428] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:05:59.565840 5766 heartbeater.cc:503] Master 127.4.177.126:38283 was elected leader, sending a full tablet report...
W20260808 21:05:59.612903 5767 tablet.cc:2369] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:05:59.701746 5623 tablet.cc:2369] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:05:59.770505 5908 tablet.cc:2369] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:05:59.879631 5935 raft_consensus.cc:482] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:05:59.879812 5935 raft_consensus.cc:504] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.880406 5935 leader_election.cc:288] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 913164c1924b45b49858468f12bc9510 (127.4.177.65:41123), 4bc93ba09acf48bcbcbec4533a103439 (127.4.177.67:40493)
I20260808 21:05:59.883499 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.880843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:44585 (local address 127.4.177.65:41123)
0808 21:05:59.880903 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.880905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.880916 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:05:59.881025 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.881026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.881055 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.881120 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:05:59.881299 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.881789 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.882769 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.883131 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.883207 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.883214 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.883365 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.883420 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 21:05:59.883490 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.880738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41123 (local address 127.4.177.66:44585)
0808 21:05:59.880847 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.880859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.880921 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.881138 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.881138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.881140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:05:59.881202 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.881203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.881872 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.881873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.882669 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.882671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.883213 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.883214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.883286 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.883291 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.883353 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.883403 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260808 21:05:59.883772 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.880735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:48045 (local address 127.4.177.67:40493)
0808 21:05:59.880792 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.880795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.880848 (+ 53us) server_negotiation.cc:407] Connection header received
0808 21:05:59.880981 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.880982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.880995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.881049 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:05:59.881252 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.881733 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.882764 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.883349 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.883413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.883421 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.883665 (+ 244us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.883716 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:05:59.883772 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.880622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40493 (local address 127.4.177.66:48045)
0808 21:05:59.880750 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 21:05:59.880766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 21:05:59.880849 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:05:59.881078 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:05:59.881079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:05:59.881082 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:05:59.881172 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.881177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.881819 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.881820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.882671 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:05:59.882673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.883426 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.883427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:05:59.883498 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.883502 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:05:59.883649 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 21:05:59.883698 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260808 21:05:59.883965 5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32102374044d2a972e3646bb341164" candidate_uuid: "2206866a221347aaa742f5b1e098fef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913164c1924b45b49858468f12bc9510" is_pre_election: true
I20260808 21:05:59.884143 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32102374044d2a972e3646bb341164" candidate_uuid: "2206866a221347aaa742f5b1e098fef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc93ba09acf48bcbcbec4533a103439" is_pre_election: true
I20260808 21:05:59.884222 5573 raft_consensus.cc:2454] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2206866a221347aaa742f5b1e098fef7 in term 0.
I20260808 21:05:59.884279 5859 raft_consensus.cc:2454] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2206866a221347aaa742f5b1e098fef7 in term 0.
I20260808 21:05:59.884719 5646 leader_election.cc:302] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2206866a221347aaa742f5b1e098fef7, 4bc93ba09acf48bcbcbec4533a103439; no voters:
I20260808 21:05:59.884941 5935 raft_consensus.cc:2781] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:05:59.884986 5935 raft_consensus.cc:482] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:05:59.885002 5935 raft_consensus.cc:3037] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:05:59.885686 5935 raft_consensus.cc:504] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.885841 5935 leader_election.cc:288] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [CANDIDATE]: Term 1 election: Requested vote from peers 913164c1924b45b49858468f12bc9510 (127.4.177.65:41123), 4bc93ba09acf48bcbcbec4533a103439 (127.4.177.67:40493)
I20260808 21:05:59.886027 5859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32102374044d2a972e3646bb341164" candidate_uuid: "2206866a221347aaa742f5b1e098fef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bc93ba09acf48bcbcbec4533a103439"
I20260808 21:05:59.886015 5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd32102374044d2a972e3646bb341164" candidate_uuid: "2206866a221347aaa742f5b1e098fef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913164c1924b45b49858468f12bc9510"
I20260808 21:05:59.886096 5859 raft_consensus.cc:3037] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:05:59.886128 5573 raft_consensus.cc:3037] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:05:59.886826 5859 raft_consensus.cc:2454] T bd32102374044d2a972e3646bb341164 P 4bc93ba09acf48bcbcbec4533a103439 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2206866a221347aaa742f5b1e098fef7 in term 1.
I20260808 21:05:59.886848 5573 raft_consensus.cc:2454] T bd32102374044d2a972e3646bb341164 P 913164c1924b45b49858468f12bc9510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2206866a221347aaa742f5b1e098fef7 in term 1.
I20260808 21:05:59.886996 5646 leader_election.cc:302] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2206866a221347aaa742f5b1e098fef7, 4bc93ba09acf48bcbcbec4533a103439; no voters:
I20260808 21:05:59.887130 5935 raft_consensus.cc:2781] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:05:59.887389 5935 raft_consensus.cc:686] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [term 1 LEADER]: Becoming Leader. State: Replica: 2206866a221347aaa742f5b1e098fef7, State: Running, Role: LEADER
I20260808 21:05:59.887502 5935 consensus_queue.cc:237] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } }
I20260808 21:05:59.888477 5415 catalog_manager.cc:5495] T bd32102374044d2a972e3646bb341164 P 2206866a221347aaa742f5b1e098fef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2206866a221347aaa742f5b1e098fef7 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "2206866a221347aaa742f5b1e098fef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2206866a221347aaa742f5b1e098fef7" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 44725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "913164c1924b45b49858468f12bc9510" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc93ba09acf48bcbcbec4533a103439" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40493 } health_report { overall_health: UNKNOWN } } }
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:05:59.908396 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.901355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.4.177.65:41123)
0808 21:05:59.901447 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.901449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.901501 (+ 52us) server_negotiation.cc:407] Connection header received
0808 21:05:59.901563 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.901564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.901574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.901625 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.902192 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.902658 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.903405 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.903589 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.903646 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.905825 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.905836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.905836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.905842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.907333 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.907662 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.907665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.907666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.907725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.907901 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.907902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.907902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.908047 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.908164 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.908221 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.908280 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:05:59.933591 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.926305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.4.177.65:41123)
0808 21:05:59.926354 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.926356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.926423 (+ 67us) server_negotiation.cc:407] Connection header received
0808 21:05:59.926469 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.926470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.926479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.926526 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.927069 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.927773 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.928561 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.928743 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.928808 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.930995 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.930997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.930997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.931005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.932342 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.932689 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.932690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.932690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.932747 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.932953 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.932954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.932955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.933058 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.933165 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.933395 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.933478 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 21:05:59 dist-test-slave-chsk krb5kdc[5365](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:05:59.957768 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:05:59.950967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.4.177.65:41123)
0808 21:05:59.951036 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:05:59.951039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:05:59.951191 (+ 152us) server_negotiation.cc:407] Connection header received
0808 21:05:59.951294 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:05:59.951295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:05:59.951313 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:05:59.951375 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:05:59.951948 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.952442 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.953172 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:05:59.953364 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:05:59.953442 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:05:59.955592 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:05:59.955594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:05:59.955594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:05:59.955601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:05:59.956773 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.957071 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.957072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.957072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.957131 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:05:59.957321 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:05:59.957322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:05:59.957323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:05:59.957429 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:05:59.957518 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 21:05:59.957605 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 21:05:59.957663 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 21:05:59.958506 5528 tablet_service.cc:2908] Rejecting scan request for tablet bd32102374044d2a972e3646bb341164: Uninitialized: clean time has not yet been initialized
I20260808 21:05:59.959509 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5480
I20260808 21:05:59.966197 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5628
I20260808 21:05:59.972369 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5771
I20260808 21:05:59.978533 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5379
2026-08-08T21:05:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2373 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:00 dist-test-slave-chsk krb5kdc[5952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:00 dist-test-slave-chsk krb5kdc[5952](info): set up 2 sockets
Aug 08 21:06:00 dist-test-slave-chsk krb5kdc[5952](info): commencing operation
krb5kdc: starting...
W20260808 21:06:01.042374 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:01.162052 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:45293
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:45293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:01.243783 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:01.244001 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:01.244024 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:01.245157 5965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:01.245199 5965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 21:06:01.245213 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:01.245226 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:01.245239 5965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:01.245253 5965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:01.246809 5965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:45293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:45293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:01.247032 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:01.247243 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:01.252239 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.252279 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.252233 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:01.252547 5965 server_base.cc:1034] running on GCE node
I20260808 21:06:01.252848 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:01.253139 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:01.254283 5965 hybrid_clock.cc:648] HybridClock initialized: now 1786223161254273 us; error 32 us; skew 500 ppm
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:01.256888 5965 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:01.257299 5965 webserver.cc:460] Webserver started at http://127.4.177.126:33363/ using document root <none> and password file <none>
I20260808 21:06:01.257539 5965 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:01.257629 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:01.257769 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:01.258775 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "80a5399c19454b498bb6bed6b55df8e7"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "e6e5fc27839eb814eb399633b79b463c"
server_key_iv: "50f9044d399b3f68b99d03c0d93060b3"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.259250 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "80a5399c19454b498bb6bed6b55df8e7"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "e6e5fc27839eb814eb399633b79b463c"
server_key_iv: "50f9044d399b3f68b99d03c0d93060b3"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.260625 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:01.261363 5980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.261524 5965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:01.261641 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "80a5399c19454b498bb6bed6b55df8e7"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "e6e5fc27839eb814eb399633b79b463c"
server_key_iv: "50f9044d399b3f68b99d03c0d93060b3"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.261714 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:01.285063 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:01.286984 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:01.287171 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:01.292042 5965 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:45293
I20260808 21:06:01.292138 6044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:45293 every 8 connection(s)
I20260808 21:06:01.292433 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:01.293080 6045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.295353 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Bootstrap starting.
I20260808 21:06:01.295965 6045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.296275 6045 log.cc:824] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:01.296937 6045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: No bootstrap required, opened a new log
I20260808 21:06:01.297080 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 5965
I20260808 21:06:01.297192 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:01.297353 4805 external_mini_cluster.cc:1428] Setting key cccfd60da9b4923ec113bc199db16c16
I20260808 21:06:01.299852 6045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } }
I20260808 21:06:01.300025 6045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.300129 6045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80a5399c19454b498bb6bed6b55df8e7, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.300293 6045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } }
I20260808 21:06:01.300375 6045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:01.300397 6045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:01.300467 6045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:01.301113 6045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } }
I20260808 21:06:01.301234 6045 leader_election.cc:302] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80a5399c19454b498bb6bed6b55df8e7; no voters:
I20260808 21:06:01.301422 6045 leader_election.cc:288] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:01.301494 6050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:01.301664 6045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:01.301669 6050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [term 1 LEADER]: Becoming Leader. State: Replica: 80a5399c19454b498bb6bed6b55df8e7, State: Running, Role: LEADER
I20260808 21:06:01.301821 6050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } }
I20260808 21:06:01.302228 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80a5399c19454b498bb6bed6b55df8e7. Latest consensus state: current_term: 1 leader_uuid: "80a5399c19454b498bb6bed6b55df8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } } }
I20260808 21:06:01.302234 6053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80a5399c19454b498bb6bed6b55df8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80a5399c19454b498bb6bed6b55df8e7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45293 } } }
I20260808 21:06:01.302330 6053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:01.302502 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7 [sys.catalog]: This master's current role is: LEADER
W20260808 21:06:01.303673 6067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:01.303728 6067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:01.303797 6061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:01.303967 6061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:01.305987 6061 catalog_manager.cc:1347] Generated new cluster ID: 017a306f22ef414cb4fb4e90e60eeece
I20260808 21:06:01.306103 6061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:01.310846 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.297870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.4.177.126:45293)
0808 21:06:01.298091 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.298107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.298139 (+ 32us) server_negotiation.cc:407] Connection header received
0808 21:06:01.299814 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.299824 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.299886 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.300055 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.300828 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.302323 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.303455 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.304061 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.304270 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.307057 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.307061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.307061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.307073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.309096 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.309498 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.309501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.309502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.309665 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.309890 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.309891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.309891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.310211 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.310375 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.310454 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.310615 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260808 21:06:01.339319 6061 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:01.339901 6061 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:01.342414 6061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Generated new TSK 0
I20260808 21:06:01.342622 6061 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:01.368309 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45293
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:01.450614 6073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:01.450811 6073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:01.450835 6073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:01.452077 6073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:01.452121 6073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:01.452142 6073 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:01.452159 6073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:01.453577 6073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:01.453815 6073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:01.454074 6073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:01.459079 6080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:01.459236 6073 server_base.cc:1034] running on GCE node
W20260808 21:06:01.459118 6079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.459121 6082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:01.459671 6073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:01.459934 6073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:01.461136 6073 hybrid_clock.cc:648] HybridClock initialized: now 1786223161461134 us; error 43 us; skew 500 ppm
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:01.463769 6073 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:01.464201 6073 webserver.cc:460] Webserver started at http://127.4.177.65:39195/ using document root <none> and password file <none>
I20260808 21:06:01.464459 6073 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:01.464530 6073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:01.464675 6073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:01.465734 6073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "5dc8ff4deea44f599ce84787b212b86b"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "95dd3cc2375365e0fbc848abe55545a5"
server_key_iv: "a5267cedbeac2a895948fbc424614f97"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.466051 6073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "5dc8ff4deea44f599ce84787b212b86b"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "95dd3cc2375365e0fbc848abe55545a5"
server_key_iv: "a5267cedbeac2a895948fbc424614f97"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.467331 6073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:06:01.468020 6091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.468199 6073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:01.468307 6073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "5dc8ff4deea44f599ce84787b212b86b"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "95dd3cc2375365e0fbc848abe55545a5"
server_key_iv: "a5267cedbeac2a895948fbc424614f97"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.468379 6073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:01.477620 6073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:01.479482 6073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:01.479647 6073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:01.480158 6073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:01.480219 6073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.480265 6073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:01.480307 6073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:01.488020 6073 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:36963
I20260808 21:06:01.488145 6217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:36963 every 8 connection(s)
I20260808 21:06:01.488438 6073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:01.491020 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.480856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:44119 (local address 127.4.177.126:45293)
0808 21:06:01.480936 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.480939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.481176 (+ 237us) server_negotiation.cc:407] Connection header received
0808 21:06:01.481893 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.481895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.481927 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.481994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.482933 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.483570 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.485262 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.485487 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.485584 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.488464 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.488466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.488466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.488477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.489835 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.490227 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.490227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.490228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.490282 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.490491 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.490492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.490492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.490601 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.490650 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.490883 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.490956 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 21:06:01.491245 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.480813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.65:44119)
0808 21:06:01.481009 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.481059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.481780 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.482109 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.482113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.482524 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.482826 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.482832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.483680 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.483681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.485154 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.485159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.485604 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.485605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.485780 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.486328 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.486340 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.488359 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.489910 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.489916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.489918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.490156 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.490340 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.490341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.490341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.490438 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.490654 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.490659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.490771 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.490853 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.490976 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260808 21:06:01.491781 6218 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45293
I20260808 21:06:01.491871 6218 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:01.492059 6218 heartbeater.cc:511] Master 127.4.177.126:45293 requested a full tablet report, sending...
I20260808 21:06:01.492660 5997 ts_manager.cc:194] Registered new tserver with Master: 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963)
I20260808 21:06:01.493386 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6073
I20260808 21:06:01.493498 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:01.493571 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:44119
I20260808 21:06:01.493697 4805 external_mini_cluster.cc:1428] Setting key bff716e81d794fcad1e26281cf7f6f8f
I20260808 21:06:01.498960 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.492897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.65:33591)
0808 21:06:01.493072 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.493082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.493149 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.493311 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.493312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.493506 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.493564 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.493566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.494351 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.494353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.495065 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.495068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.495448 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.495449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.495537 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.495926 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.495932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.496330 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.498093 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.498094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.498094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.498250 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.498444 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.498445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.498445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.498488 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.498734 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.498734 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.498767 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.498812 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.498884 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260808 21:06:01.499063 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.492982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:33591 (local address 127.4.177.126:45293)
0808 21:06:01.493027 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.493029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.493151 (+ 122us) server_negotiation.cc:407] Connection header received
0808 21:06:01.493226 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.493227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.493239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.493308 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.493643 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.494251 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.495188 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.495374 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.495427 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.496418 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.496419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.496419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.496425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.498012 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.498321 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.498322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.498322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.498383 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.498540 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.498541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.498541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.498675 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.498746 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.498835 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.498923 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:01.521438 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45293
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:01.605047 6227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:01.605276 6227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:01.605315 6227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:01.606474 6227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:01.606539 6227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:01.606577 6227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:01.606616 6227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:01.608107 6227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:01.608373 6227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:01.608608 6227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:01.613672 6235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.613685 6232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.613684 6233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:01.613723 6227 server_base.cc:1034] running on GCE node
I20260808 21:06:01.614078 6227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:01.614319 6227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:01.615477 6227 hybrid_clock.cc:648] HybridClock initialized: now 1786223161615469 us; error 40 us; skew 500 ppm
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:01.618053 6227 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:01.618489 6227 webserver.cc:460] Webserver started at http://127.4.177.66:38017/ using document root <none> and password file <none>
I20260808 21:06:01.618718 6227 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:01.618788 6227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:01.618939 6227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:01.620029 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "27421368362342a9a4cf6d75ae402bc2"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "9991aca283d8d974df59ab9158d5d77e"
server_key_iv: "a7fc434b00e51cfc1e4b81a95a7091e5"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.620396 6227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "27421368362342a9a4cf6d75ae402bc2"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "9991aca283d8d974df59ab9158d5d77e"
server_key_iv: "a7fc434b00e51cfc1e4b81a95a7091e5"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.621876 6227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:01.622697 6243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.622886 6227 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:01.622992 6227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "27421368362342a9a4cf6d75ae402bc2"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "9991aca283d8d974df59ab9158d5d77e"
server_key_iv: "a7fc434b00e51cfc1e4b81a95a7091e5"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.623080 6227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:01.641599 6227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:01.643880 6227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:01.644068 6227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:01.644644 6227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:01.644686 6227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.644716 6227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:01.644732 6227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.652122 6227 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:39945
I20260808 21:06:01.652200 6365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:39945 every 8 connection(s)
I20260808 21:06:01.652566 6227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:01.656191 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.645267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:40949 (local address 127.4.177.126:45293)
0808 21:06:01.645322 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.645324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.645582 (+ 258us) server_negotiation.cc:407] Connection header received
0808 21:06:01.646178 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.646179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.646193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.646258 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.647323 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.648203 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.649903 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.650204 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.650260 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.653267 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.653269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.653269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.653276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.654704 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.655102 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.655104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.655104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.655235 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.655528 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.655530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.655530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.655700 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.655778 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.656039 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.656124 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:01.656335 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.645256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.66:40949)
0808 21:06:01.645450 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.645487 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.646072 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.646383 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.646388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.646854 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.647198 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.647205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.648310 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.648311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.649782 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.649786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.650297 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.650299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.650471 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.651115 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.651128 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.653179 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.654786 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.654793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.654794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.655013 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.655308 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.655309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.655309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.655413 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.655782 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.655788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.655914 (+ 126us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.656006 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.656135 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260808 21:06:01.656848 6366 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45293
I20260808 21:06:01.656931 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6227
I20260808 21:06:01.656961 6366 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:01.657035 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:01.657169 6366 heartbeater.cc:511] Master 127.4.177.126:45293 requested a full tablet report, sending...
I20260808 21:06:01.657230 4805 external_mini_cluster.cc:1428] Setting key b3bb8688a9f2f35ef57381bb72fffd54
I20260808 21:06:01.657574 5996 ts_manager.cc:194] Registered new tserver with Master: 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66:39945)
I20260808 21:06:01.658118 5996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:40949
I20260808 21:06:01.664855 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.657972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.66:55167)
0808 21:06:01.658124 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.658133 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.658206 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.658366 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.658367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.658574 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.658641 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.658644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.659585 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.659586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.660449 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.660452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.660790 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.660791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.660875 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.661365 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.661369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.661660 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.663701 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.663702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.663702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.663865 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.664096 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.664097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.664097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.664217 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.664605 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.664606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.664646 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.664701 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.664778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260808 21:06:01.664912 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.658011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:55167 (local address 127.4.177.126:45293)
0808 21:06:01.658053 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.658055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.658193 (+ 138us) server_negotiation.cc:407] Connection header received
0808 21:06:01.658292 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.658293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.658305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.658363 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.658726 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.659477 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.660537 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.660716 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.660769 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.661847 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.661849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.661849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.661857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.663610 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.663941 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.663943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.663943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.664010 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.664290 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.664291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.664291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.664535 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.664617 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.664730 (+ 113us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.664810 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:01.685747 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45293
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:40989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:01.772073 6375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:01.772287 6375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:01.772327 6375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:01.773464 6375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:01.773537 6375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:01.773574 6375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:01.773631 6375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:01.775046 6375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:40989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:01.775326 6375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:01.775569 6375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:01.780303 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.780323 6384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:01.780304 6382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:01.780700 6375 server_base.cc:1034] running on GCE node
I20260808 21:06:01.780968 6375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:01.781257 6375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:01.782435 6375 hybrid_clock.cc:648] HybridClock initialized: now 1786223161782421 us; error 26 us; skew 500 ppm
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:01.785131 6375 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:01.785548 6375 webserver.cc:460] Webserver started at http://127.4.177.67:44333/ using document root <none> and password file <none>
I20260808 21:06:01.785796 6375 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:01.785871 6375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:01.786026 6375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:01.787048 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "058ab53361a843c3bf41b7cd9daf5c65"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "f1686cbc122727d5f93d2bb8b17f76d0"
server_key_iv: "766a5dac6c1bef7e19033ac21831b36e"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.787489 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "058ab53361a843c3bf41b7cd9daf5c65"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "f1686cbc122727d5f93d2bb8b17f76d0"
server_key_iv: "766a5dac6c1bef7e19033ac21831b36e"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.788839 6375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:01.789577 6391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.789744 6375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:01.789844 6375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "058ab53361a843c3bf41b7cd9daf5c65"
format_stamp: "Formatted at 2026-08-08 21:06:01 on dist-test-slave-chsk"
server_key: "f1686cbc122727d5f93d2bb8b17f76d0"
server_key_iv: "766a5dac6c1bef7e19033ac21831b36e"
server_key_version: "encryptionkey@0"
I20260808 21:06:01.789925 6375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:01.803617 6375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:01.805483 6375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:01.805644 6375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:01.806175 6375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:01.806238 6375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.806282 6375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:01.806311 6375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:01.814570 6375 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40079
I20260808 21:06:01.814669 6518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40079 every 8 connection(s)
I20260808 21:06:01.814989 6375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:01.816738 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.806640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:41379 (local address 127.4.177.126:45293)
0808 21:06:01.806710 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.806713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.806876 (+ 163us) server_negotiation.cc:407] Connection header received
0808 21:06:01.807424 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.807425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.807440 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.807528 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.808165 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.809020 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.810697 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.811027 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.811121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.813671 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.813672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.813673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.813681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.815532 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.816000 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.816001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.816001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.816056 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.816266 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.816266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.816266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.816361 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.816407 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.816617 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.816676 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:06:01.816967 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.806613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.67:41379)
0808 21:06:01.806753 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.806791 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.807329 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.807614 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.807619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.807914 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.808090 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.808095 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.809141 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.809142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.810590 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.810593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.811169 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.811170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.811326 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.811869 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.811880 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.813575 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.815602 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.815605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.815607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.815929 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.816130 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.816130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.816131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.816224 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.816411 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.816415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.816520 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.816601 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.816716 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260808 21:06:01.817405 6519 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45293
I20260808 21:06:01.817512 6519 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:01.817698 6519 heartbeater.cc:511] Master 127.4.177.126:45293 requested a full tablet report, sending...
I20260808 21:06:01.818096 5997 ts_manager.cc:194] Registered new tserver with Master: 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079)
I20260808 21:06:01.818720 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:41379
I20260808 21:06:01.821224 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6375
I20260808 21:06:01.821317 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:01.821516 4805 external_mini_cluster.cc:1428] Setting key db424696380d0dffd31701929b555cfa
I20260808 21:06:01.822407 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:06:01.824272 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.818495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33049 (local address 127.4.177.126:45293)
0808 21:06:01.818864 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.818866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.818882 (+ 16us) server_negotiation.cc:407] Connection header received
0808 21:06:01.818916 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.818916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.818926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.818986 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.819370 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.819835 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.820531 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.820738 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.820813 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.821808 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.821810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.821810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.821817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.823209 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.823596 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.823597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.823597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.823652 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.823861 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.823861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.823861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.823950 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.823997 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.824161 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.824216 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260808 21:06:01.824415 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.818422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.4.177.67:33049)
0808 21:06:01.818632 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.818641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.818684 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.819006 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.819006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.819226 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:01.819286 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.819287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.819926 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.819927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.820447 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.820449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.820812 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.820813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.820893 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.821269 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:01.821274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:01.821704 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:01.823291 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.823292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.823293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.823521 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.823718 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:01.823719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:01.823719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:01.823790 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:01.824009 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:01.824011 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:01.824061 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.824145 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.824258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":149,"threads_started":1}
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 21:06:01 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:01.847316 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.840201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.4.177.126:45293)
0808 21:06:01.840335 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.840337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.840349 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:01.840397 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.840398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.840408 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.840465 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:01.841030 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.841538 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.842343 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.842646 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.842706 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.844943 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:01.844945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:01.844945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:01.844951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:01.846215 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.846542 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.846543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.846543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.846595 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:01.846805 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:01.846805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:01.846806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:01.846904 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:01.846978 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.847088 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.847214 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260808 21:06:01.849227 5997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:01.850771 5997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:01.857750 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40079 (local address 127.0.0.1:39698)
0808 21:06:01.853573 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.853583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.853626 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.854099 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.854103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.854105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:01.854175 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.854177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855104 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.855105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.856450 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.856453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.857393 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.857394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.857489 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.857503 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.857599 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.857659 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":81,"threads_started":1}
I20260808 21:06:01.857769 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.4.177.67:40079)
0808 21:06:01.853842 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.853845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.853853 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:01.853886 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.853887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.853924 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.854003 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:01.854270 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855012 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.856519 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.857296 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.857374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.857432 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.857618 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.857663 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":202,"threads_started":1}
I20260808 21:06:01.858042 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.4.177.65:36963)
0808 21:06:01.854086 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.854090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.854100 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:01.854137 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.854139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.854153 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.854262 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:01.854371 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855500 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.856810 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.857591 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.857686 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.857741 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.857859 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.857899 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":91,"threads_started":1}
I20260808 21:06:01.857956 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36963 (local address 127.0.0.1:47912)
0808 21:06:01.853316 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.853359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.853427 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.854268 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.854269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.854271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:01.854313 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.854315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855595 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.855596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.856727 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.856748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.857691 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.857692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.857758 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.857762 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.857833 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.857880 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":32,"thread_start_us":91,"threads_started":1}
I20260808 21:06:01.858513 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.4.177.66:39945)
0808 21:06:01.853696 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.853700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.853954 (+ 254us) server_negotiation.cc:407] Connection header received
0808 21:06:01.854026 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.854029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.854046 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.854137 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:01.854370 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855517 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.856730 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.857283 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.857401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.857462 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.858372 (+ 910us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.858411 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260808 21:06:01.858461 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.853429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:39945 (local address 127.0.0.1:51716)
0808 21:06:01.853882 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.853893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.853954 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.854225 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.854226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.854228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:01.854285 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.854287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.855611 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.855612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.856631 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.856633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.858221 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.858223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.858277 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.858281 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.858354 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.858395 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260808 21:06:01.859378 6145 tablet_service.cc:1467] Processing CreateTablet for tablet 5940ad82622643f0b413ec457f90b440 (DEFAULT_TABLE table=test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.859449 6297 tablet_service.cc:1467] Processing CreateTablet for tablet 5940ad82622643f0b413ec457f90b440 (DEFAULT_TABLE table=test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.859499 6449 tablet_service.cc:1467] Processing CreateTablet for tablet 5940ad82622643f0b413ec457f90b440 (DEFAULT_TABLE table=test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.859802 6145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5940ad82622643f0b413ec457f90b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.859802 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5940ad82622643f0b413ec457f90b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.859958 6449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5940ad82622643f0b413ec457f90b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.862671 6544 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Bootstrap starting.
I20260808 21:06:01.863622 6545 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Bootstrap starting.
I20260808 21:06:01.863683 6544 tablet_bootstrap.cc:654] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.864125 6544 log.cc:824] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:01.864334 6545 tablet_bootstrap.cc:654] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.864738 6545 log.cc:824] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:01.865314 6544 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: No bootstrap required, opened a new log
I20260808 21:06:01.865391 6544 ts_tablet_manager.cc:1397] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260808 21:06:01.865474 6545 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: No bootstrap required, opened a new log
I20260808 21:06:01.865540 6545 ts_tablet_manager.cc:1397] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:01.865810 6549 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Bootstrap starting.
I20260808 21:06:01.866449 6549 tablet_bootstrap.cc:654] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.866868 6549 log.cc:824] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:01.867324 6545 raft_consensus.cc:348] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.867512 6545 raft_consensus.cc:374] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.867560 6545 raft_consensus.cc:729] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27421368362342a9a4cf6d75ae402bc2, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.867684 6545 consensus_queue.cc:260] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.867733 6549 tablet_bootstrap.cc:492] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: No bootstrap required, opened a new log
I20260808 21:06:01.867650 6544 raft_consensus.cc:348] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.867807 6549 ts_tablet_manager.cc:1397] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:01.867800 6544 raft_consensus.cc:374] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.867846 6544 raft_consensus.cc:729] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058ab53361a843c3bf41b7cd9daf5c65, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.867972 6545 ts_tablet_manager.cc:1428] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:01.867986 6544 consensus_queue.cc:260] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.868106 6366 heartbeater.cc:503] Master 127.4.177.126:45293 was elected leader, sending a full tablet report...
I20260808 21:06:01.868263 6544 ts_tablet_manager.cc:1428] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:06:01.868288 6519 heartbeater.cc:503] Master 127.4.177.126:45293 was elected leader, sending a full tablet report...
I20260808 21:06:01.869731 6549 raft_consensus.cc:348] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.869865 6549 raft_consensus.cc:374] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.869887 6549 raft_consensus.cc:729] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc8ff4deea44f599ce84787b212b86b, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.869959 6549 consensus_queue.cc:260] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.870143 6549 ts_tablet_manager.cc:1428] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:01.870244 6218 heartbeater.cc:503] Master 127.4.177.126:45293 was elected leader, sending a full tablet report...
W20260808 21:06:01.903272 6367 tablet.cc:2369] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:01.911046 6551 raft_consensus.cc:482] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:01.911237 6551 raft_consensus.cc:504] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.911659 6551 leader_election.cc:288] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079)
I20260808 21:06:01.915311 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.911826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40079 (local address 127.4.177.66:41629)
0808 21:06:01.911871 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.911884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.911932 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.912122 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.912124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.912126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:01.912201 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.912203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.912904 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.912906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.914192 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.914194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.914980 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.914981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.915071 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.915076 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.915164 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.915212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260808 21:06:01.915623 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.911917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:41629 (local address 127.4.177.67:40079)
0808 21:06:01.911971 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.911974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.911985 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:01.912018 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.912019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.912030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.912102 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:01.912274 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.912821 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.914284 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.914853 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.914931 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.914943 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.915498 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.915552 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:01.916770 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.911885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:46651 (local address 127.4.177.65:36963)
0808 21:06:01.912021 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.912022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.912922 (+ 900us) server_negotiation.cc:407] Connection header received
0808 21:06:01.912954 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.912955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.912966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.913020 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:01.913232 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.913985 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.915588 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.916248 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.916317 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.916326 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.916643 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.916693 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 21:06:01.916786 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.911829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36963 (local address 127.4.177.66:46651)
0808 21:06:01.912206 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:01.912218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:01.912276 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:01.913049 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:01.913050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:01.913053 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:01.913139 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.913142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.914096 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.914098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.915464 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:01.915467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.916431 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.916433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:01.916528 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.916532 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:01.916591 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 21:06:01.916656 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":203,"thread_start_us":120,"threads_started":1}
I20260808 21:06:01.917277 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5940ad82622643f0b413ec457f90b440" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b" is_pre_election: true
I20260808 21:06:01.917479 6167 raft_consensus.cc:2454] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 0.
I20260808 21:06:01.918536 6244 leader_election.cc:302] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27421368362342a9a4cf6d75ae402bc2, 5dc8ff4deea44f599ce84787b212b86b; no voters:
I20260808 21:06:01.918681 6551 raft_consensus.cc:2781] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:01.918672 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5940ad82622643f0b413ec457f90b440" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058ab53361a843c3bf41b7cd9daf5c65" is_pre_election: true
I20260808 21:06:01.918742 6551 raft_consensus.cc:482] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:01.918771 6551 raft_consensus.cc:3037] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:01.918831 6469 raft_consensus.cc:2454] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 0.
I20260808 21:06:01.919991 6551 raft_consensus.cc:504] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.920142 6551 leader_election.cc:288] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079)
I20260808 21:06:01.920285 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5940ad82622643f0b413ec457f90b440" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b"
I20260808 21:06:01.920320 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5940ad82622643f0b413ec457f90b440" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058ab53361a843c3bf41b7cd9daf5c65"
I20260808 21:06:01.920387 6469 raft_consensus.cc:3037] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:01.920377 6167 raft_consensus.cc:3037] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:01.921290 6167 raft_consensus.cc:2454] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 1.
I20260808 21:06:01.921290 6469 raft_consensus.cc:2454] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 1.
I20260808 21:06:01.921442 6244 leader_election.cc:302] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27421368362342a9a4cf6d75ae402bc2, 5dc8ff4deea44f599ce84787b212b86b; no voters:
I20260808 21:06:01.921532 6551 raft_consensus.cc:2781] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:01.921681 6551 raft_consensus.cc:686] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 27421368362342a9a4cf6d75ae402bc2, State: Running, Role: LEADER
I20260808 21:06:01.921785 6551 consensus_queue.cc:237] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.922636 5997 catalog_manager.cc:5495] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:01.933797 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:01.930700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.4.177.66:39945)
0808 21:06:01.930759 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:01.930760 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:01.930770 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:01.930806 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:01.930807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:01.930823 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:01.930902 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:01.931081 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.931615 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.932715 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:01.933028 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:01.933103 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:01.933370 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:01.933503 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:01.933604 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:01.933670 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 21:06:01.933696 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:06:01.935832 6167 raft_consensus.cc:1261] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Refusing update from remote peer 27421368362342a9a4cf6d75ae402bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:01.936266 6555 consensus_queue.cc:1035] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:01.938474 6469 raft_consensus.cc:1261] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Refusing update from remote peer 27421368362342a9a4cf6d75ae402bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:01.938879 6551 consensus_queue.cc:1035] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:01.939463 6559 mvcc.cc:205] Tried to move back new op lower bound from 7316370071285895168 to 7316370071232200704. Current Snapshot: MvccSnapshot[applied={T|T < 7316370071285895168}]
I20260808 21:06:01.943729 5997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:01.943876 5997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:01.944875 5997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Sending DeleteTablet for 3 replicas of tablet 5940ad82622643f0b413ec457f90b440
I20260808 21:06:01.945443 6449 tablet_service.cc:1514] Processing DeleteTablet for tablet 5940ad82622643f0b413ec457f90b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:01 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:39698
I20260808 21:06:01.945463 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5940ad82622643f0b413ec457f90b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:01 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260808 21:06:01.945508 6145 tablet_service.cc:1514] Processing DeleteTablet for tablet 5940ad82622643f0b413ec457f90b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:01 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47912
I20260808 21:06:01.945731 6569 tablet_replica.cc:330] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: stopping tablet replica
I20260808 21:06:01.945837 6571 tablet_replica.cc:330] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: stopping tablet replica
I20260808 21:06:01.945849 6569 raft_consensus.cc:2229] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:01.945962 6569 raft_consensus.cc:2258] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:01.945909 5997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:01.946048 5997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:01.946204 6571 raft_consensus.cc:2229] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:01.946491 6569 ts_tablet_manager.cc:1905] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:01.946727 6571 raft_consensus.cc:2258] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:01.947443 6449 tablet_service.cc:1467] Processing CreateTablet for tablet ef12381688b040e8ac48513322eca6fc (DEFAULT_TABLE table=test-table [id=2784cb976a6d42299b814fe1b243aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.947551 6145 tablet_service.cc:1467] Processing CreateTablet for tablet ef12381688b040e8ac48513322eca6fc (DEFAULT_TABLE table=test-table [id=2784cb976a6d42299b814fe1b243aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.947582 6449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef12381688b040e8ac48513322eca6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.947671 6145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef12381688b040e8ac48513322eca6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.948884 6569 ts_tablet_manager.cc:1918] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:01.948943 6544 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Bootstrap starting.
I20260808 21:06:01.948951 6569 log.cc:1196] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/5940ad82622643f0b413ec457f90b440
I20260808 21:06:01.949048 6549 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Bootstrap starting.
I20260808 21:06:01.949290 6569 ts_tablet_manager.cc:1939] T 5940ad82622643f0b413ec457f90b440 P 058ab53361a843c3bf41b7cd9daf5c65: Deleting consensus metadata
I20260808 21:06:01.949396 6571 ts_tablet_manager.cc:1905] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:01.949438 6544 tablet_bootstrap.cc:654] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.949470 6549 tablet_bootstrap.cc:654] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.949832 6297 tablet_service.cc:1467] Processing CreateTablet for tablet ef12381688b040e8ac48513322eca6fc (DEFAULT_TABLE table=test-table [id=2784cb976a6d42299b814fe1b243aa93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:01.949788 5983 catalog_manager.cc:4841] TS 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079): tablet 5940ad82622643f0b413ec457f90b440 (table test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]) successfully deleted
I20260808 21:06:01.949954 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef12381688b040e8ac48513322eca6fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:01.950376 6549 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: No bootstrap required, opened a new log
I20260808 21:06:01.950431 6549 ts_tablet_manager.cc:1397] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:01.950601 6549 raft_consensus.cc:348] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.950695 6549 raft_consensus.cc:374] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.950719 6549 raft_consensus.cc:729] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc8ff4deea44f599ce84787b212b86b, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.950770 6549 consensus_queue.cc:260] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.950875 6549 ts_tablet_manager.cc:1428] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:01.951346 6545 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Bootstrap starting.
I20260808 21:06:01.951823 6545 tablet_bootstrap.cc:654] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:01.951818 6570 tablet_replica.cc:330] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: stopping tablet replica
I20260808 21:06:01.952831 6545 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: No bootstrap required, opened a new log
I20260808 21:06:01.952904 6545 ts_tablet_manager.cc:1397] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:01.953202 6545 raft_consensus.cc:348] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.953332 6545 raft_consensus.cc:374] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.953363 6545 raft_consensus.cc:729] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27421368362342a9a4cf6d75ae402bc2, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.953451 6545 consensus_queue.cc:260] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.954101 6545 ts_tablet_manager.cc:1428] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:01.952571 6570 raft_consensus.cc:2229] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:01.954602 6570 raft_consensus.cc:2258] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:01.954756 6571 ts_tablet_manager.cc:1918] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:01.954890 6571 log.cc:1196] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/5940ad82622643f0b413ec457f90b440
I20260808 21:06:01.955468 6570 ts_tablet_manager.cc:1905] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:01.955479 6571 ts_tablet_manager.cc:1939] T 5940ad82622643f0b413ec457f90b440 P 5dc8ff4deea44f599ce84787b212b86b: Deleting consensus metadata
I20260808 21:06:01.956017 5981 catalog_manager.cc:4841] TS 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963): tablet 5940ad82622643f0b413ec457f90b440 (table test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]) successfully deleted
I20260808 21:06:01.957499 6570 ts_tablet_manager.cc:1918] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:01.957573 6570 log.cc:1196] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/5940ad82622643f0b413ec457f90b440
I20260808 21:06:01.957840 6570 ts_tablet_manager.cc:1939] T 5940ad82622643f0b413ec457f90b440 P 27421368362342a9a4cf6d75ae402bc2: Deleting consensus metadata
I20260808 21:06:01.958459 5981 catalog_manager.cc:4841] TS 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66:39945): tablet 5940ad82622643f0b413ec457f90b440 (table test-table [id=01a57d0ce31b4e4282c86ed86b7569dc]) successfully deleted
I20260808 21:06:01.963647 6544 tablet_bootstrap.cc:492] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: No bootstrap required, opened a new log
I20260808 21:06:01.963754 6544 ts_tablet_manager.cc:1397] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.000s
I20260808 21:06:01.963940 6544 raft_consensus.cc:348] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.964048 6544 raft_consensus.cc:374] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:01.964078 6544 raft_consensus.cc:729] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058ab53361a843c3bf41b7cd9daf5c65, State: Initialized, Role: FOLLOWER
I20260808 21:06:01.964128 6544 consensus_queue.cc:260] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:01.964239 6544 ts_tablet_manager.cc:1428] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260808 21:06:01.989449 6219 tablet.cc:2369] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:02.046674 6552 raft_consensus.cc:482] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:02.046799 6552 raft_consensus.cc:504] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.047166 6552 leader_election.cc:288] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66:39945)
I20260808 21:06:02.051123 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.047894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:41893 (local address 127.4.177.66:39945)
0808 21:06:02.047949 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.047951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.047961 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:02.047989 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.047990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.048001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.048066 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:02.048251 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.048993 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050105 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.050704 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050786 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.050796 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.051008 (+ 212us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.051051 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:06:02.051355 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.047432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36963 (local address 127.4.177.67:42245)
0808 21:06:02.047501 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.047589 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.047653 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.047898 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.047900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.047902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:02.047974 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.047976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.049029 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.049030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.050189 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.050193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.050991 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.051098 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.051102 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.051219 (+ 117us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.051271 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 21:06:02.051489 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.047671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:39945 (local address 127.4.177.67:41893)
0808 21:06:02.047847 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.047859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.047912 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.048084 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.048084 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.048086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:02.048156 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.048158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.049093 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.049095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.049993 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.049996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.050825 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.050900 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.050905 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.051293 (+ 388us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.051391 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":115,"threads_started":1}
I20260808 21:06:02.051858 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.047541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:42245 (local address 127.4.177.65:36963)
0808 21:06:02.047591 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.047593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.047637 (+ 44us) server_negotiation.cc:407] Connection header received
0808 21:06:02.047733 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.047734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.047745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.047825 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:02.048048 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.048917 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050295 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.050900 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.050991 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.051000 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.051536 (+ 536us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.051592 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:02.051924 6317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef12381688b040e8ac48513322eca6fc" candidate_uuid: "058ab53361a843c3bf41b7cd9daf5c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27421368362342a9a4cf6d75ae402bc2" is_pre_election: true
I20260808 21:06:02.052064 6317 raft_consensus.cc:2454] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058ab53361a843c3bf41b7cd9daf5c65 in term 0.
I20260808 21:06:02.052095 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef12381688b040e8ac48513322eca6fc" candidate_uuid: "058ab53361a843c3bf41b7cd9daf5c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b" is_pre_election: true
I20260808 21:06:02.052176 6167 raft_consensus.cc:2454] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 058ab53361a843c3bf41b7cd9daf5c65 in term 0.
I20260808 21:06:02.052371 6392 leader_election.cc:302] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058ab53361a843c3bf41b7cd9daf5c65, 5dc8ff4deea44f599ce84787b212b86b; no voters:
I20260808 21:06:02.052515 6552 raft_consensus.cc:2781] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:02.052596 6552 raft_consensus.cc:482] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:02.052621 6552 raft_consensus.cc:3037] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.053534 6552 raft_consensus.cc:504] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.053676 6552 leader_election.cc:288] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66:39945)
I20260808 21:06:02.053936 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef12381688b040e8ac48513322eca6fc" candidate_uuid: "058ab53361a843c3bf41b7cd9daf5c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b"
I20260808 21:06:02.053961 6317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef12381688b040e8ac48513322eca6fc" candidate_uuid: "058ab53361a843c3bf41b7cd9daf5c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27421368362342a9a4cf6d75ae402bc2"
I20260808 21:06:02.054075 6317 raft_consensus.cc:3037] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.054080 6167 raft_consensus.cc:3037] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.054858 6167 raft_consensus.cc:2454] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058ab53361a843c3bf41b7cd9daf5c65 in term 1.
I20260808 21:06:02.055048 6392 leader_election.cc:302] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058ab53361a843c3bf41b7cd9daf5c65, 5dc8ff4deea44f599ce84787b212b86b; no voters:
I20260808 21:06:02.055087 6317 raft_consensus.cc:2454] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 058ab53361a843c3bf41b7cd9daf5c65 in term 1.
I20260808 21:06:02.055197 6552 raft_consensus.cc:2781] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:02.055419 6552 raft_consensus.cc:686] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 LEADER]: Becoming Leader. State: Replica: 058ab53361a843c3bf41b7cd9daf5c65, State: Running, Role: LEADER
I20260808 21:06:02.055509 6552 consensus_queue.cc:237] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.056401 5996 catalog_manager.cc:5495] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "058ab53361a843c3bf41b7cd9daf5c65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } health_report { overall_health: HEALTHY } } }
W20260808 21:06:02.065961 6520 tablet.cc:2369] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 21:06:02 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:02.100337 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.088056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45293 (local address 127.0.0.1:54672)
0808 21:06:02.088333 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.088346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.088463 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.088635 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.088636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.089002 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:02.089070 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.089073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.089811 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.089812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.090398 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.090399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.090816 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.090817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.090886 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.095963 (+ 5077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:02.095970 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:02.097551 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:02.099133 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:02.099163 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:02.099165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:02.099373 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:02.099615 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:02.099615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:02.099616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:02.099672 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:02.099968 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:02.099976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:02.100053 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.100128 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.100245 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":144,"threads_started":1}
I20260808 21:06:02.100394 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.088153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.4.177.126:45293)
0808 21:06:02.088300 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.088303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.088445 (+ 142us) server_negotiation.cc:407] Connection header received
0808 21:06:02.088527 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.088528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.088547 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.088629 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:02.089174 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.089719 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.090493 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.090701 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.090799 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.097667 (+ 6868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:02.097668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:02.097669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:02.097676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:02.099032 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.099460 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.099461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.099462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.099544 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.099728 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.099728 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.099729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.099847 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:02.099951 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.100160 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.100298 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":105}
I20260808 21:06:02.103271 5996 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54672:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 21:06:02.103592 5996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:02.105170 6145 tablet_service.cc:1467] Processing CreateTablet for tablet 351ff0c919844f6a9bcca86e249d36e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcbf45046fb54ccb94abbcec8dcc4d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:02.105187 6449 tablet_service.cc:1467] Processing CreateTablet for tablet 351ff0c919844f6a9bcca86e249d36e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcbf45046fb54ccb94abbcec8dcc4d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:02.105170 6297 tablet_service.cc:1467] Processing CreateTablet for tablet 351ff0c919844f6a9bcca86e249d36e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bcbf45046fb54ccb94abbcec8dcc4d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:02.105397 6449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351ff0c919844f6a9bcca86e249d36e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:02.105397 6297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351ff0c919844f6a9bcca86e249d36e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:02.105398 6145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351ff0c919844f6a9bcca86e249d36e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:02.106892 6545 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: Bootstrap starting.
I20260808 21:06:02.106894 6549 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: Bootstrap starting.
I20260808 21:06:02.107316 6545 tablet_bootstrap.cc:654] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:02.107360 6549 tablet_bootstrap.cc:654] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:02.107560 6544 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: Bootstrap starting.
I20260808 21:06:02.107935 6549 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: No bootstrap required, opened a new log
I20260808 21:06:02.107969 6549 ts_tablet_manager.cc:1397] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:02.107995 6544 tablet_bootstrap.cc:654] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:02.108114 6549 raft_consensus.cc:348] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.108165 6549 raft_consensus.cc:374] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:02.108177 6549 raft_consensus.cc:729] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dc8ff4deea44f599ce84787b212b86b, State: Initialized, Role: FOLLOWER
I20260808 21:06:02.108238 6549 consensus_queue.cc:260] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.108354 6549 ts_tablet_manager.cc:1428] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:02.108398 6553 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.108449 6545 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: No bootstrap required, opened a new log
I20260808 21:06:02.108469 6553 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.108498 6545 ts_tablet_manager.cc:1397] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:02.108615 6545 raft_consensus.cc:348] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.108678 6545 raft_consensus.cc:374] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:02.108692 6545 raft_consensus.cc:729] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27421368362342a9a4cf6d75ae402bc2, State: Initialized, Role: FOLLOWER
I20260808 21:06:02.108727 6545 consensus_queue.cc:260] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.108824 6545 ts_tablet_manager.cc:1428] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:02.108843 6551 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.108919 6551 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.109202 6544 tablet_bootstrap.cc:492] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: No bootstrap required, opened a new log
I20260808 21:06:02.109257 6544 ts_tablet_manager.cc:1397] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260808 21:06:02.109407 6544 raft_consensus.cc:348] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.109483 6544 raft_consensus.cc:374] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:02.109514 6544 raft_consensus.cc:729] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 058ab53361a843c3bf41b7cd9daf5c65, State: Initialized, Role: FOLLOWER
I20260808 21:06:02.109565 6544 consensus_queue.cc:260] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.109655 6544 ts_tablet_manager.cc:1428] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:02.109720 6552 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.109812 6552 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.206223 6551 raft_consensus.cc:482] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:02.206378 6551 raft_consensus.cc:504] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.206614 6551 leader_election.cc:288] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079)
I20260808 21:06:02.206820 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351ff0c919844f6a9bcca86e249d36e2" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b" is_pre_election: true
I20260808 21:06:02.206835 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351ff0c919844f6a9bcca86e249d36e2" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058ab53361a843c3bf41b7cd9daf5c65" is_pre_election: true
I20260808 21:06:02.206935 6469 raft_consensus.cc:2454] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 0.
I20260808 21:06:02.206935 6167 raft_consensus.cc:2454] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 0.
I20260808 21:06:02.207118 6247 leader_election.cc:302] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 058ab53361a843c3bf41b7cd9daf5c65, 27421368362342a9a4cf6d75ae402bc2; no voters:
I20260808 21:06:02.207297 6551 raft_consensus.cc:2781] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:02.207341 6551 raft_consensus.cc:482] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:02.207362 6551 raft_consensus.cc:3037] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.208089 6551 raft_consensus.cc:504] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.208236 6551 leader_election.cc:288] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 election: Requested vote from peers 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963), 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079)
I20260808 21:06:02.208436 6167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351ff0c919844f6a9bcca86e249d36e2" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dc8ff4deea44f599ce84787b212b86b"
I20260808 21:06:02.208470 6469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351ff0c919844f6a9bcca86e249d36e2" candidate_uuid: "27421368362342a9a4cf6d75ae402bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "058ab53361a843c3bf41b7cd9daf5c65"
I20260808 21:06:02.208530 6167 raft_consensus.cc:3037] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.208536 6469 raft_consensus.cc:3037] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:02.209354 6167 raft_consensus.cc:2454] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 1.
I20260808 21:06:02.209571 6244 leader_election.cc:302] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27421368362342a9a4cf6d75ae402bc2, 5dc8ff4deea44f599ce84787b212b86b; no voters:
I20260808 21:06:02.209661 6469 raft_consensus.cc:2454] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27421368362342a9a4cf6d75ae402bc2 in term 1.
I20260808 21:06:02.209700 6551 raft_consensus.cc:2781] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:02.209777 6551 raft_consensus.cc:686] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 27421368362342a9a4cf6d75ae402bc2, State: Running, Role: LEADER
I20260808 21:06:02.209833 6551 consensus_queue.cc:237] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } }
I20260808 21:06:02.210147 6555 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: TxnStatusTablet state changed. Reason: New leader 27421368362342a9a4cf6d75ae402bc2. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.210264 6555 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:02.210445 6589 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 21:06:02.210538 5996 catalog_manager.cc:5495] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:02.212085 6589 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 21:06:02.226524 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.223906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:39945 (local address 127.0.0.1:51724)
0808 21:06:02.224012 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.224025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.224080 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.224224 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.224224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.224226 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:02.224280 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.224283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.224914 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.224916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.225628 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.225630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.226023 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.226024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.226093 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.226121 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.226336 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.226347 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.226402 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.226451 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 21:06:02.226511 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.223973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.4.177.66:39945)
0808 21:06:02.224044 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.224047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.224058 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:02.224161 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.224162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.224173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.224219 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:02.224341 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.224831 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.225734 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.225929 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.226021 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.226234 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.226273 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.226334 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.226407 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.226430 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:02.227708 6469 raft_consensus.cc:1261] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Refusing update from remote peer 27421368362342a9a4cf6d75ae402bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:02.227902 6167 raft_consensus.cc:1261] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Refusing update from remote peer 27421368362342a9a4cf6d75ae402bc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:02.227969 6551 consensus_queue.cc:1035] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:02.228091 6555 consensus_queue.cc:1035] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:02.228638 6552 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: TxnStatusTablet state changed. Reason: New leader 27421368362342a9a4cf6d75ae402bc2. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.228753 6552 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.229362 6555 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.229451 6555 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:02.229789 6553 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: TxnStatusTablet state changed. Reason: New leader 27421368362342a9a4cf6d75ae402bc2. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.229884 6553 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.230069 6558 mvcc.cc:205] Tried to move back new op lower bound from 7316370072482930688 to 7316370072412057600. Current Snapshot: MvccSnapshot[applied={T|T < 7316370072482930688 or (T in {7316370072482930688})}]
I20260808 21:06:02.230259 6565 mvcc.cc:205] Tried to move back new op lower bound from 7316370072482930688 to 7316370072412057600. Current Snapshot: MvccSnapshot[applied={T|T < 7316370072482930688}]
I20260808 21:06:02.230325 6552 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.230398 6552 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 058ab53361a843c3bf41b7cd9daf5c65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.230374 6555 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.230451 6555 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 27421368362342a9a4cf6d75ae402bc2: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:02.231473 6553 tablet_replica.cc:439] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27421368362342a9a4cf6d75ae402bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 } } peers { permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 } } peers { permanent_uuid: "058ab53361a843c3bf41b7cd9daf5c65" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40079 } } }
I20260808 21:06:02.231585 6553 tablet_replica.cc:442] T 351ff0c919844f6a9bcca86e249d36e2 P 5dc8ff4deea44f599ce84787b212b86b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:02.234743 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.232370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.4.177.67:40079)
0808 21:06:02.232441 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.232443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.232451 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:02.232484 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.232485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.232512 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.232590 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:02.232715 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.233230 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.233929 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.234146 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.234221 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.234394 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.234509 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.234595 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.234641 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.234660 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 21:06:02.240144 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.237478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:39945 (local address 127.4.177.67:55579)
0808 21:06:02.237570 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.237581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.237618 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.237729 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.237730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.237731 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:02.237796 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.237797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.238369 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.238370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.239339 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.239342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.239721 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.239722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.239790 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.239806 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.240004 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.240006 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.240049 (+ 43us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.240088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 21:06:02.240152 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.237556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:55579 (local address 127.4.177.66:39945)
0808 21:06:02.237615 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.237617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.237625 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:02.237671 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.237672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.237681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.237726 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:02.237859 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.238291 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.239417 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.239648 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.239701 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.239906 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.239942 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.239993 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.240052 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.240075 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:02.242645 6167 raft_consensus.cc:1261] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Refusing update from remote peer 058ab53361a843c3bf41b7cd9daf5c65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:02.242753 6317 raft_consensus.cc:1261] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Refusing update from remote peer 058ab53361a843c3bf41b7cd9daf5c65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:02.242909 6552 consensus_queue.cc:1035] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dc8ff4deea44f599ce84787b212b86b" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:02.243026 6576 consensus_queue.cc:1035] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27421368362342a9a4cf6d75ae402bc2" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:02.252086 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.249216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:39907 (local address 127.4.177.67:40079)
0808 21:06:02.249308 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.249310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.249692 (+ 382us) server_negotiation.cc:407] Connection header received
0808 21:06:02.249751 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.249752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.249760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.249805 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:02.249914 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.250366 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.251476 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.251663 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.251718 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.251897 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.251922 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.251959 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.252012 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.252029 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 21:06:02.252355 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.249165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40079 (local address 127.4.177.66:39907)
0808 21:06:02.249477 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:02.249489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:02.249692 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:02.249809 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:02.249810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:02.249812 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:02.249860 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.249862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.250443 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.250444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.251396 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:02.251399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.251738 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.251739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:02.251800 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.251818 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:02.251960 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:02.251962 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:02.252011 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 21:06:02.252053 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":88,"threads_started":1}
I20260808 21:06:02.261015 5995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:02.261138 5995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:02.261833 5995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80a5399c19454b498bb6bed6b55df8e7: Sending DeleteTablet for 3 replicas of tablet ef12381688b040e8ac48513322eca6fc
I20260808 21:06:02.262290 6297 tablet_service.cc:1514] Processing DeleteTablet for tablet ef12381688b040e8ac48513322eca6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:02 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260808 21:06:02.262408 6570 tablet_replica.cc:330] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: stopping tablet replica
I20260808 21:06:02.262497 6570 raft_consensus.cc:2229] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:02.262550 6570 raft_consensus.cc:2258] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:02.263252 6145 tablet_service.cc:1514] Processing DeleteTablet for tablet ef12381688b040e8ac48513322eca6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:02 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47912
I20260808 21:06:02.263358 6571 tablet_replica.cc:330] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: stopping tablet replica
I20260808 21:06:02.263411 6571 raft_consensus.cc:2229] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:02.263453 6571 raft_consensus.cc:2258] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:02.264662 6571 ts_tablet_manager.cc:1905] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:02.264659 6570 ts_tablet_manager.cc:1905] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:02.266458 6570 ts_tablet_manager.cc:1918] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:02.266459 6449 tablet_service.cc:1514] Processing DeleteTablet for tablet ef12381688b040e8ac48513322eca6fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:02 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:39698
I20260808 21:06:02.266520 6570 log.cc:1196] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/ef12381688b040e8ac48513322eca6fc
I20260808 21:06:02.266570 6569 tablet_replica.cc:330] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: stopping tablet replica
I20260808 21:06:02.266611 6569 raft_consensus.cc:2229] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:02.266709 6569 raft_consensus.cc:2258] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:02.266752 6570 ts_tablet_manager.cc:1939] T ef12381688b040e8ac48513322eca6fc P 27421368362342a9a4cf6d75ae402bc2: Deleting consensus metadata
I20260808 21:06:02.267025 6569 ts_tablet_manager.cc:1905] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:02.267117 5981 catalog_manager.cc:4841] TS 27421368362342a9a4cf6d75ae402bc2 (127.4.177.66:39945): tablet ef12381688b040e8ac48513322eca6fc (table test-table [id=2784cb976a6d42299b814fe1b243aa93]) successfully deleted
I20260808 21:06:02.268646 6569 ts_tablet_manager.cc:1918] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:02.268712 6569 log.cc:1196] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/ef12381688b040e8ac48513322eca6fc
I20260808 21:06:02.268929 6569 ts_tablet_manager.cc:1939] T ef12381688b040e8ac48513322eca6fc P 058ab53361a843c3bf41b7cd9daf5c65: Deleting consensus metadata
I20260808 21:06:02.269254 5983 catalog_manager.cc:4841] TS 058ab53361a843c3bf41b7cd9daf5c65 (127.4.177.67:40079): tablet ef12381688b040e8ac48513322eca6fc (table test-table [id=2784cb976a6d42299b814fe1b243aa93]) successfully deleted
I20260808 21:06:02.269296 6571 ts_tablet_manager.cc:1918] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:02.269354 6571 log.cc:1196] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/ef12381688b040e8ac48513322eca6fc
I20260808 21:06:02.269585 6571 ts_tablet_manager.cc:1939] T ef12381688b040e8ac48513322eca6fc P 5dc8ff4deea44f599ce84787b212b86b: Deleting consensus metadata
I20260808 21:06:02.269909 5981 catalog_manager.cc:4841] TS 5dc8ff4deea44f599ce84787b212b86b (127.4.177.65:36963): tablet ef12381688b040e8ac48513322eca6fc (table test-table [id=2784cb976a6d42299b814fe1b243aa93]) successfully deleted
Aug 08 21:06:02 dist-test-slave-chsk krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223161, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:06:02.288036 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.264568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.4.177.65:36963)
0808 21:06:02.264609 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.264611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.275009 (+ 10398us) server_negotiation.cc:407] Connection header received
0808 21:06:02.275226 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.275227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.275240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.275274 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:02.276280 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.277042 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.278225 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.278680 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.278756 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.283871 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:02.283887 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:02.283887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:02.283896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:02.286586 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.287013 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.287018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.287018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.287122 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.287394 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.287395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.287396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.287601 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:02.287756 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.287833 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.287928 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 21:06:02.288347 6212 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47928
I20260808 21:06:02.297143 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:02.289447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.4.177.126:45293)
0808 21:06:02.289505 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:02.289507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:02.289571 (+ 64us) server_negotiation.cc:407] Connection header received
0808 21:06:02.289643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:02.289644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:02.289656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:02.289714 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:02.290328 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.291123 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.292281 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:02.292607 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:02.292650 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:02.294201 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:02.294203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:02.294203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:02.294211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:02.295852 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.296260 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.296261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.296261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.296320 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:02.296517 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:02.296518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:02.296518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:02.296646 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:02.296722 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:02.296937 (+ 215us) server_negotiation.cc:299] Negotiation successful
0808 21:06:02.297042 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260808 21:06:02.297610 5995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54674
I20260808 21:06:02.298350 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6073
I20260808 21:06:02.309788 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6227
I20260808 21:06:02.317536 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6375
I20260808 21:06:02.325145 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 5965
2026-08-08T21:06:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2347 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:02 dist-test-slave-chsk krb5kdc[6611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:02 dist-test-slave-chsk krb5kdc[6611](info): set up 2 sockets
Aug 08 21:06:02 dist-test-slave-chsk krb5kdc[6611](info): commencing operation
krb5kdc: starting...
W20260808 21:06:03.398964 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:03.517684 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33249
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:03.600384 6624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:03.600608 6624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:03.600646 6624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:03.601775 6624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:03.601838 6624 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 21:06:03.601861 6624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:03.601894 6624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:03.601922 6624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:03.601945 6624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:03.603514 6624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:33249
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:03.603776 6624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:03.604027 6624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:03.608702 6629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:03.608768 6632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:03.608698 6630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:03.608913 6624 server_base.cc:1034] running on GCE node
I20260808 21:06:03.609254 6624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:03.609542 6624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:03.610693 6624 hybrid_clock.cc:648] HybridClock initialized: now 1786223163610680 us; error 45 us; skew 500 ppm
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:03.613303 6624 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:03.613675 6624 webserver.cc:460] Webserver started at http://127.4.177.126:44757/ using document root <none> and password file <none>
I20260808 21:06:03.613871 6624 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:03.613910 6624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:03.613996 6624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:03.615037 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "8da7b948306a41bd80ba30767fda906e"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "1f603e21e289f8290e4ec0c4128a69ab"
server_key_iv: "e976ee19a84472e1b50d393422ac704c"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.615417 6624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "8da7b948306a41bd80ba30767fda906e"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "1f603e21e289f8290e4ec0c4128a69ab"
server_key_iv: "e976ee19a84472e1b50d393422ac704c"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.616683 6624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:03.617349 6640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:03.617522 6624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:03.617621 6624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "8da7b948306a41bd80ba30767fda906e"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "1f603e21e289f8290e4ec0c4128a69ab"
server_key_iv: "e976ee19a84472e1b50d393422ac704c"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.617702 6624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:03.627491 6624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:03.629328 6624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:03.629487 6624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:03.634337 6624 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:33249
I20260808 21:06:03.634414 6706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:33249 every 8 connection(s)
I20260808 21:06:03.634742 6624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:03.635468 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:03.637668 6707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Bootstrap starting.
I20260808 21:06:03.638218 6707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:03.638455 6707 log.cc:824] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:03.639125 6707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: No bootstrap required, opened a new log
I20260808 21:06:03.641435 6707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } }
I20260808 21:06:03.641577 6707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:03.641600 6707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da7b948306a41bd80ba30767fda906e, State: Initialized, Role: FOLLOWER
I20260808 21:06:03.641736 6707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } }
I20260808 21:06:03.641826 6707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:03.641850 6707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:03.641880 6707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:03.642409 6707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } }
I20260808 21:06:03.642495 6707 leader_election.cc:302] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da7b948306a41bd80ba30767fda906e; no voters:
I20260808 21:06:03.642633 6707 leader_election.cc:288] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:03.642726 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:03.642903 6707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:03.642894 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6624
I20260808 21:06:03.642983 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:03.642935 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [term 1 LEADER]: Becoming Leader. State: Replica: 8da7b948306a41bd80ba30767fda906e, State: Running, Role: LEADER
I20260808 21:06:03.643198 4805 external_mini_cluster.cc:1428] Setting key 354a140bc8a3d2032464eaee38a04381
I20260808 21:06:03.643165 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } }
I20260808 21:06:03.643566 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da7b948306a41bd80ba30767fda906e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } } }
I20260808 21:06:03.643661 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:03.644240 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da7b948306a41bd80ba30767fda906e. Latest consensus state: current_term: 1 leader_uuid: "8da7b948306a41bd80ba30767fda906e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da7b948306a41bd80ba30767fda906e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33249 } } }
I20260808 21:06:03.644351 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e [sys.catalog]: This master's current role is: LEADER
W20260808 21:06:03.644951 6730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:03.645007 6730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:03.645098 6720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:03.645334 6720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:03.647027 6720 catalog_manager.cc:1347] Generated new cluster ID: 6c4170ca031d4b9ca51fe4cee413fda3
I20260808 21:06:03.647078 6720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:03.656138 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.643894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.4.177.126:33249)
0808 21:06:03.644046 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:03.644056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:03.644583 (+ 527us) server_negotiation.cc:407] Connection header received
0808 21:06:03.644847 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:03.644857 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:03.644921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:03.645112 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:03.646026 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.647345 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.648393 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.648994 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.649101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.652070 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:03.652080 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:03.652081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:03.652093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:03.654066 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.654503 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.654505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.654506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.654612 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.654829 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.654830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.654830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.655131 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:03.655298 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:03.655717 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 21:06:03.655851 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260808 21:06:03.658504 6720 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:03.659266 6720 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:03.670172 6720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Generated new TSK 0
I20260808 21:06:03.670377 6720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:03.701506 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33249
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:03.787714 6735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:03.787953 6735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:03.787997 6735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:03.789143 6735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:03.789216 6735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:03.789251 6735 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 21:06:03.789283 6735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:03.789316 6735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:03.790771 6735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:03.791047 6735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:03.791307 6735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:03.796370 6740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:03.796378 6741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:03.796366 6743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:03.796622 6735 server_base.cc:1034] running on GCE node
I20260808 21:06:03.796967 6735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:03.797209 6735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:03.798355 6735 hybrid_clock.cc:648] HybridClock initialized: now 1786223163798333 us; error 36 us; skew 500 ppm
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:03.800933 6735 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:03.801362 6735 webserver.cc:460] Webserver started at http://127.4.177.65:32883/ using document root <none> and password file <none>
I20260808 21:06:03.801613 6735 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:03.801684 6735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:03.801832 6735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:03.802886 6735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "ae1b1463ff52404c88f103d77f11e9f3"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "9e531ef6598a78bad5ea04e02d39cde0"
server_key_iv: "b3797bb9c9ecf91f861c6a070e47f415"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.803301 6735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "ae1b1463ff52404c88f103d77f11e9f3"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "9e531ef6598a78bad5ea04e02d39cde0"
server_key_iv: "b3797bb9c9ecf91f861c6a070e47f415"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.804976 6735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:03.805761 6750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:03.805935 6735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:03.806056 6735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "ae1b1463ff52404c88f103d77f11e9f3"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "9e531ef6598a78bad5ea04e02d39cde0"
server_key_iv: "b3797bb9c9ecf91f861c6a070e47f415"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.806134 6735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:03.825253 6735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:03.827731 6735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:03.827907 6735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:03.828454 6735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:03.828521 6735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:03.828572 6735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:03.828606 6735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:03.836395 6735 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:33199
I20260808 21:06:03.836427 6875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:33199 every 8 connection(s)
I20260808 21:06:03.836861 6735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:03.837208 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6735
I20260808 21:06:03.837311 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:03.837505 4805 external_mini_cluster.cc:1428] Setting key b47934dc73a05290ffc02eca0713e7ca
I20260808 21:06:03.839100 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.829016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:47707 (local address 127.4.177.126:33249)
0808 21:06:03.829098 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:03.829101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:03.829364 (+ 263us) server_negotiation.cc:407] Connection header received
0808 21:06:03.830142 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:03.830143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:03.830153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:03.830204 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:03.831175 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.831760 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.832761 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.832987 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.833077 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.835794 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:03.835800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:03.835800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:03.835808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:03.837574 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.838071 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.838072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.838072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.838140 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.838441 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.838442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.838442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.838565 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:03.838619 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:03.838954 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 21:06:03.839034 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260808 21:06:03.839412 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.828996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.65:47707)
0808 21:06:03.829209 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:03.829262 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:03.829999 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:03.830286 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:03.830291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:03.830710 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:03.831013 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.831021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.831862 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.831863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.832662 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.832665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.833090 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.833091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.833261 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.833786 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:03.833794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:03.835684 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:03.837656 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.837665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.837667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.837965 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.838201 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.838202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.838202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.838364 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.838633 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:03.838640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:03.838810 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 21:06:03.838931 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 21:06:03.839121 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260808 21:06:03.839895 6876 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33249
I20260808 21:06:03.839977 6876 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:03.840159 6876 heartbeater.cc:511] Master 127.4.177.126:33249 requested a full tablet report, sending...
I20260808 21:06:03.840677 6658 ts_manager.cc:194] Registered new tserver with Master: ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199)
I20260808 21:06:03.841532 6658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:47707
I20260808 21:06:03.847294 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.840929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.65:47049)
0808 21:06:03.841091 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:03.841103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:03.841154 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:03.841302 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:03.841303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:03.841490 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:03.841551 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.841554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.842492 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.842492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.843087 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.843088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.843700 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.843701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.843756 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.844217 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:03.844223 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:03.844522 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:03.846316 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.846317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.846317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.846476 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.846692 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.846692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.846692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.846741 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.847011 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:03.847012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:03.847045 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 21:06:03.847095 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:06:03.847223 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260808 21:06:03.847482 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.841113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:47049 (local address 127.4.177.126:33249)
0808 21:06:03.841152 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:03.841154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:03.841163 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:03.841219 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:03.841220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:03.841230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:03.841291 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:03.841670 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.842391 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.843255 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.843598 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.843686 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.844604 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:03.844605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:03.844606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:03.844614 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:03.846231 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.846557 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.846559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.846559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.846623 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.846803 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.846804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.846804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.846945 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:03.847017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:03.847176 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 21:06:03.847263 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:03.866098 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33249
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:03.947968 6887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:03.948172 6887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:03.948199 6887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:03.949324 6887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:03.949368 6887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:03.949388 6887 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 21:06:03.949399 6887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:03.949416 6887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:03.950779 6887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:03.950958 6887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:03.951161 6887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:03.956373 6894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:03.956498 6887 server_base.cc:1034] running on GCE node
W20260808 21:06:03.956391 6895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:03.956598 6897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:03.956787 6887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:03.957026 6887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:03.958199 6887 hybrid_clock.cc:648] HybridClock initialized: now 1786223163958181 us; error 31 us; skew 500 ppm
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:03.960599 6887 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:03.961052 6887 webserver.cc:460] Webserver started at http://127.4.177.66:44303/ using document root <none> and password file <none>
I20260808 21:06:03.961283 6887 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:03.961352 6887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:03.961493 6887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:03.962507 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "35cb56fabac64cdf969e29a38c15f7a0"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "68dd25eb6e30e33da26f97c92c6f6017"
server_key_iv: "2f53c78b8a0c4fca79cf3594bc21ddf6"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.962885 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "35cb56fabac64cdf969e29a38c15f7a0"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "68dd25eb6e30e33da26f97c92c6f6017"
server_key_iv: "2f53c78b8a0c4fca79cf3594bc21ddf6"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.964249 6887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 21:06:03.964944 6904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:03.965129 6887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:03.965226 6887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "35cb56fabac64cdf969e29a38c15f7a0"
format_stamp: "Formatted at 2026-08-08 21:06:03 on dist-test-slave-chsk"
server_key: "68dd25eb6e30e33da26f97c92c6f6017"
server_key_iv: "2f53c78b8a0c4fca79cf3594bc21ddf6"
server_key_version: "encryptionkey@0"
I20260808 21:06:03.965312 6887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:03.974807 6887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:03.976564 6887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:03.976715 6887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:03.977245 6887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:03.977309 6887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:03.977350 6887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:03.977378 6887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:03 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:03.984761 6887 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37161
I20260808 21:06:03.984841 7026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37161 every 8 connection(s)
I20260808 21:06:03.985122 6887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:03.987589 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.977600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:55323 (local address 127.4.177.126:33249)
0808 21:06:03.977668 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:03.977677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:03.977914 (+ 237us) server_negotiation.cc:407] Connection header received
0808 21:06:03.978399 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:03.978401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:03.978413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:03.978476 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:03.979280 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.979806 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.980782 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.981093 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.981164 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.984722 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:03.984723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:03.984723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:03.984732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:03.986246 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.986697 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.986698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.986699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.986765 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.987023 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.987024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.987024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.987131 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:03.987233 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:03.987459 (+ 226us) server_negotiation.cc:299] Negotiation successful
0808 21:06:03.987526 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:06:03.987741 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.977586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.66:55323)
0808 21:06:03.977780 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:03.977818 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:03.978297 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:03.978544 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:03.978546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:03.978880 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:03.979200 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.979205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.979912 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.979913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.980693 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.980695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.981197 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.981199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.981386 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.981950 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:03.981961 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:03.984588 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:03.986391 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.986397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.986399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.986619 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.986837 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.986838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.986838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.986962 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.987226 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:03.987233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:03.987346 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 21:06:03.987430 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 21:06:03.987548 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":129,"threads_started":1}
I20260808 21:06:03.988262 7027 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33249
I20260808 21:06:03.988360 7027 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:03.988554 7027 heartbeater.cc:511] Master 127.4.177.126:33249 requested a full tablet report, sending...
I20260808 21:06:03.988934 6659 ts_manager.cc:194] Registered new tserver with Master: 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161)
I20260808 21:06:03.989396 6659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:55323
I20260808 21:06:03.991115 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 6887
I20260808 21:06:03.991250 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:03.991432 4805 external_mini_cluster.cc:1428] Setting key 42f70fc1441ac9178845bde306454a3d
I20260808 21:06:03.995111 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.989336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:45633 (local address 127.4.177.126:33249)
0808 21:06:03.989376 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:03.989378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:03.989474 (+ 96us) server_negotiation.cc:407] Connection header received
0808 21:06:03.989530 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:03.989531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:03.989540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:03.989600 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:03.989893 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.990412 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.991162 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.991429 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.991513 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.992440 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:03.992441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:03.992442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:03.992448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:03.993855 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.994208 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.994210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.994210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.994265 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:03.994489 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:03.994490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:03.994490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:03.994644 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:03.994721 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:03.994900 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 21:06:03.994992 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:03.995119 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:03.989251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.66:45633)
0808 21:06:03.989395 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:03.989406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:03.989470 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:03.989603 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:03.989604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:03.989762 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:03.989815 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.989817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.990508 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.990509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.991052 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:03.991054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:03.991515 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:03.991516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:03.991594 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:03.992065 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:03.992070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:03.992353 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:03.993958 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.993959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.993959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.994136 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.994345 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:03.994346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:03.994347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:03.994421 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:03.994712 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:03.994713 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:03.994764 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 21:06:03.994839 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 21:06:03.994951 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:04.015971 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33249
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:04.097115 7037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:04.097324 7037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:04.097349 7037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:04.098496 7037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:04.098539 7037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:04.098560 7037 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 21:06:04.098618 7037 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:04.098649 7037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:04.100121 7037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:33249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:04.100363 7037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:04.100610 7037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:04.105536 7046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:04.105643 7043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:04.105825 7044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:04.105800 7037 server_base.cc:1034] running on GCE node
I20260808 21:06:04.106052 7037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:04.106302 7037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:04.107462 7037 hybrid_clock.cc:648] HybridClock initialized: now 1786223164107449 us; error 48 us; skew 500 ppm
Aug 08 21:06:04 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:04.110086 7037 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:04.110620 7037 webserver.cc:460] Webserver started at http://127.4.177.67:41751/ using document root <none> and password file <none>
I20260808 21:06:04.110819 7037 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:04.110862 7037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:04.110954 7037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:04.112043 7037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "4f5dc9d0f138468b8d0f742558e0848c"
format_stamp: "Formatted at 2026-08-08 21:06:04 on dist-test-slave-chsk"
server_key: "c780032278bf8df1f5fc28e141517ad8"
server_key_iv: "c90747a4914bc977c263510c52e17d47"
server_key_version: "encryptionkey@0"
I20260808 21:06:04.112358 7037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "4f5dc9d0f138468b8d0f742558e0848c"
format_stamp: "Formatted at 2026-08-08 21:06:04 on dist-test-slave-chsk"
server_key: "c780032278bf8df1f5fc28e141517ad8"
server_key_iv: "c90747a4914bc977c263510c52e17d47"
server_key_version: "encryptionkey@0"
I20260808 21:06:04.113636 7037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:04.114321 7053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.114475 7037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:04.114569 7037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "4f5dc9d0f138468b8d0f742558e0848c"
format_stamp: "Formatted at 2026-08-08 21:06:04 on dist-test-slave-chsk"
server_key: "c780032278bf8df1f5fc28e141517ad8"
server_key_iv: "c90747a4914bc977c263510c52e17d47"
server_key_version: "encryptionkey@0"
I20260808 21:06:04.114624 7037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:04.147655 7037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:04.149600 7037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:04.149724 7037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:04.150211 7037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:04.150247 7037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.150274 7037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:04.150290 7037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:04 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223164, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:04.158041 7037 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:41273
I20260808 21:06:04.158095 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:41273 every 8 connection(s)
I20260808 21:06:04.158392 7037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:04.161343 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7037
I20260808 21:06:04.161440 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:04.161530 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.150813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33999 (local address 127.4.177.126:33249)
0808 21:06:04.150877 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.150880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.151212 (+ 332us) server_negotiation.cc:407] Connection header received
0808 21:06:04.151795 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.151797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.151814 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.151906 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:04.152769 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.153580 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.155226 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.155597 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.155695 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.158553 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:04.158554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:04.158554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:04.158563 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:04.160123 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.160535 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.160537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.160537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.160600 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.160855 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.160856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.160856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.160977 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:04.161053 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.161343 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.161437 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 21:06:04.161639 4805 external_mini_cluster.cc:1428] Setting key edaa29085295a7dbdfd602cb6b7b50f2
I20260808 21:06:04.161690 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.150807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.67:33999)
0808 21:06:04.151032 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.151081 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.151640 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.151978 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.151983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.152356 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:04.152647 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.152654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.153724 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.153725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.155054 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.155062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.155710 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.155712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.155869 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.156479 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:04.156500 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:04.158454 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:04.160227 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.160233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.160236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.160457 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.160669 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.160670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.160670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.160785 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.161044 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:04.161051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:04.161202 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.161305 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.161459 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260808 21:06:04.162258 7178 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33249
I20260808 21:06:04.162513 7178 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:04.162730 7178 heartbeater.cc:511] Master 127.4.177.126:33249 requested a full tablet report, sending...
I20260808 21:06:04.163084 6657 ts_manager.cc:194] Registered new tserver with Master: 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:04.163617 6657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:33999
I20260808 21:06:04.168826 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.163393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:34947 (local address 127.4.177.126:33249)
0808 21:06:04.163451 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.163453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.163597 (+ 144us) server_negotiation.cc:407] Connection header received
0808 21:06:04.163622 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.163623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.163634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.163694 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:04.164028 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.164486 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.165150 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.165356 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.165440 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.166311 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:04.166313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:04.166313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:04.166320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:04.167675 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.168065 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.168066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.168066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.168119 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.168327 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.168328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.168328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.168437 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:04.168508 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.168663 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.168734 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:06:04.168839 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.163350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.4.177.67:34947)
0808 21:06:04.163526 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.163535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.163587 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.163745 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.163746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.163901 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:04.163954 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.163955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.164580 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.164580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.165068 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.165069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.165441 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.165442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.165520 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.165931 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:04.165936 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:04.166208 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:04.167777 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.167778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.167778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.167991 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.168206 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.168207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.168207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.168267 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.168512 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:04.168513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:04.168556 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.168637 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.168736 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":134,"threads_started":1}
I20260808 21:06:04.173290 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 21:06:04 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 21:06:04 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:04.197013 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.189823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.4.177.126:33249)
0808 21:06:04.189888 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.189890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.189898 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:04.189931 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.189932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.189941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.190001 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:04.190572 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.191055 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.191825 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.192020 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.192100 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.194381 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:04.194382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:04.194383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:04.194390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:04.195850 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.196223 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.196224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.196224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.196285 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.196495 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.196496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.196496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.196615 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:04.196724 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.196820 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.196906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 21:06:04.198679 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:04.200374 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:04.207206 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.202967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37161 (local address 127.0.0.1:36908)
0808 21:06:04.203508 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.203516 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.203575 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.203909 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.203910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.203911 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.203961 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.203962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.204817 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.204818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.206049 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.206052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.206839 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.206840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.206918 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.206933 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.207036 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.207095 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":108,"threads_started":1}
I20260808 21:06:04.207216 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.203256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.4.177.66:37161)
0808 21:06:04.203446 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.203451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.203710 (+ 259us) server_negotiation.cc:407] Connection header received
0808 21:06:04.203759 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.203761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.203777 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.203856 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.204054 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.204719 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.206158 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.206750 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.206839 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.206903 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.207055 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.207100 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260808 21:06:04.207440 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.202959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:41273 (local address 127.0.0.1:47150)
0808 21:06:04.203357 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.203369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.203427 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.203724 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.203727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.203730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.203812 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.203815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.205191 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.205192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.206157 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.206160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.207130 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.207132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.207272 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.207276 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.207308 (+ 32us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.207359 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":12,"thread_start_us":85,"threads_started":1}
I20260808 21:06:04.207828 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.203175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.4.177.67:41273)
0808 21:06:04.203404 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.203410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.203422 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:04.203501 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.203503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.203519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.203895 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.203957 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.205058 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.206262 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.207019 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.207547 (+ 528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.207614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.207643 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.207716 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260808 21:06:04.207887 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.202959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33199 (local address 127.0.0.1:45996)
0808 21:06:04.203121 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.203228 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.203297 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.203758 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.203758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.203760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.203812 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.203815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.205163 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.205164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.206263 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.206266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.207132 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.207133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.207231 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.207235 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.207765 (+ 530us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.207815 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260808 21:06:04.207927 6957 tablet_service.cc:1467] Processing CreateTablet for tablet b72cf6818ae94aacb98a285292b7d404 (DEFAULT_TABLE table=test-table [id=0902617e80be4bd690a996edbe7436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.208259 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.203174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.4.177.65:33199)
0808 21:06:04.203448 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.203451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.203460 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:04.203501 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.203503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.203519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.203895 (+ 376us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.203957 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.205070 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.206343 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.207023 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.207993 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.208063 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.208090 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.208137 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260808 21:06:04.208372 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72cf6818ae94aacb98a285292b7d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.208869 7105 tablet_service.cc:1467] Processing CreateTablet for tablet b72cf6818ae94aacb98a285292b7d404 (DEFAULT_TABLE table=test-table [id=0902617e80be4bd690a996edbe7436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.208915 6803 tablet_service.cc:1467] Processing CreateTablet for tablet b72cf6818ae94aacb98a285292b7d404 (DEFAULT_TABLE table=test-table [id=0902617e80be4bd690a996edbe7436a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.209193 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72cf6818ae94aacb98a285292b7d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.209239 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72cf6818ae94aacb98a285292b7d404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.211921 7201 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Bootstrap starting.
I20260808 21:06:04.212294 7202 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Bootstrap starting.
I20260808 21:06:04.212814 7204 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Bootstrap starting.
I20260808 21:06:04.213001 7202 tablet_bootstrap.cc:654] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.213029 7201 tablet_bootstrap.cc:654] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.213399 7201 log.cc:824] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:04.213399 7202 log.cc:824] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:04.213562 7204 tablet_bootstrap.cc:654] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.213850 7204 log.cc:824] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:04.214210 7201 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: No bootstrap required, opened a new log
I20260808 21:06:04.214284 7201 ts_tablet_manager.cc:1397] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:04.214449 7202 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: No bootstrap required, opened a new log
I20260808 21:06:04.214515 7202 ts_tablet_manager.cc:1397] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:04.214609 7204 tablet_bootstrap.cc:492] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: No bootstrap required, opened a new log
I20260808 21:06:04.214671 7204 ts_tablet_manager.cc:1397] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:04.216033 7202 raft_consensus.cc:348] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.216193 7202 raft_consensus.cc:374] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.216238 7202 raft_consensus.cc:729] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.216349 7202 consensus_queue.cc:260] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.216470 7201 raft_consensus.cc:348] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.216629 7201 raft_consensus.cc:374] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.216655 7202 ts_tablet_manager.cc:1428] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:04.216668 7201 raft_consensus.cc:729] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.216619 7204 raft_consensus.cc:348] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.216758 7027 heartbeater.cc:503] Master 127.4.177.126:33249 was elected leader, sending a full tablet report...
I20260808 21:06:04.216755 7204 raft_consensus.cc:374] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.216794 7204 raft_consensus.cc:729] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b1463ff52404c88f103d77f11e9f3, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.216809 7201 consensus_queue.cc:260] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.216897 7204 consensus_queue.cc:260] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.217125 7178 heartbeater.cc:503] Master 127.4.177.126:33249 was elected leader, sending a full tablet report...
I20260808 21:06:04.217217 7204 ts_tablet_manager.cc:1428] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 21:06:04.217386 6876 heartbeater.cc:503] Master 127.4.177.126:33249 was elected leader, sending a full tablet report...
I20260808 21:06:04.217737 7201 ts_tablet_manager.cc:1428] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:04.225116 7209 raft_consensus.cc:482] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:04.225222 7209 raft_consensus.cc:504] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.225556 7209 leader_election.cc:288] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161)
I20260808 21:06:04.229341 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.225765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:56249 (local address 127.4.177.65:33199)
0808 21:06:04.225811 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.225813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.225824 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:04.225925 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.225926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.225937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.226001 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.226263 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.227006 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.228422 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.228973 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.229051 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.229058 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.229235 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.229274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:04.229341 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.225886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:40511 (local address 127.4.177.66:37161)
0808 21:06:04.225956 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.225958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.226146 (+ 188us) server_negotiation.cc:407] Connection header received
0808 21:06:04.226199 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.226200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.226207 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.226275 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.226410 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.227084 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.228421 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.228974 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.229051 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.229058 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.229235 (+ 177us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.229274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:06:04.229411 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.225708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33199 (local address 127.4.177.67:56249)
0808 21:06:04.225767 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.225778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.225819 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.226109 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.226111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.226113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.226188 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.226189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.227079 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.227079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.228332 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.228335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.229059 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.229060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.229162 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.229165 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.229217 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.229246 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":36}
I20260808 21:06:04.229485 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.225718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37161 (local address 127.4.177.67:40511)
0808 21:06:04.225886 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.225896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.226146 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.226289 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.226290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.226292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.226349 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.226350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.227190 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.227191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.228333 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.228335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.229059 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.229060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.229133 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.229138 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.229199 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.229240 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":13,"thread_start_us":90,"threads_started":1}
I20260808 21:06:04.229777 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72cf6818ae94aacb98a285292b7d404" candidate_uuid: "4f5dc9d0f138468b8d0f742558e0848c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3" is_pre_election: true
I20260808 21:06:04.229818 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72cf6818ae94aacb98a285292b7d404" candidate_uuid: "4f5dc9d0f138468b8d0f742558e0848c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cb56fabac64cdf969e29a38c15f7a0" is_pre_election: true
I20260808 21:06:04.229928 6977 raft_consensus.cc:2454] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dc9d0f138468b8d0f742558e0848c in term 0.
I20260808 21:06:04.229928 6824 raft_consensus.cc:2454] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5dc9d0f138468b8d0f742558e0848c in term 0.
I20260808 21:06:04.230166 7057 leader_election.cc:302] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, 4f5dc9d0f138468b8d0f742558e0848c; no voters:
I20260808 21:06:04.230296 7209 raft_consensus.cc:2781] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:04.230369 7209 raft_consensus.cc:482] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:04.230387 7209 raft_consensus.cc:3037] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.231125 7209 raft_consensus.cc:504] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.231343 7209 leader_election.cc:288] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [CANDIDATE]: Term 1 election: Requested vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161)
I20260808 21:06:04.231467 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72cf6818ae94aacb98a285292b7d404" candidate_uuid: "4f5dc9d0f138468b8d0f742558e0848c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3"
I20260808 21:06:04.231478 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72cf6818ae94aacb98a285292b7d404" candidate_uuid: "4f5dc9d0f138468b8d0f742558e0848c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cb56fabac64cdf969e29a38c15f7a0"
I20260808 21:06:04.231550 6977 raft_consensus.cc:3037] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.231540 6824 raft_consensus.cc:3037] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.232229 6824 raft_consensus.cc:2454] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dc9d0f138468b8d0f742558e0848c in term 1.
I20260808 21:06:04.232303 6977 raft_consensus.cc:2454] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5dc9d0f138468b8d0f742558e0848c in term 1.
I20260808 21:06:04.232400 7055 leader_election.cc:302] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5dc9d0f138468b8d0f742558e0848c, ae1b1463ff52404c88f103d77f11e9f3; no voters:
I20260808 21:06:04.232486 7209 raft_consensus.cc:2781] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:04.232690 7209 raft_consensus.cc:686] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 LEADER]: Becoming Leader. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Running, Role: LEADER
I20260808 21:06:04.232793 7209 consensus_queue.cc:237] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } }
I20260808 21:06:04.233569 6657 catalog_manager.cc:5495] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67). New cstate: current_term: 1 leader_uuid: "4f5dc9d0f138468b8d0f742558e0848c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } health_report { overall_health: UNKNOWN } } }
W20260808 21:06:04.235646 7028 tablet.cc:2369] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 21:06:04 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223163, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:04.283021 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.276005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.4.177.126:33249)
0808 21:06:04.276071 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.276073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.276258 (+ 185us) server_negotiation.cc:407] Connection header received
0808 21:06:04.276346 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.276347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.276359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.276428 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:04.276940 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.277444 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.278146 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.278334 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.278420 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.280643 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:04.280645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:04.280645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:04.280651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:04.282076 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.282406 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.282406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.282407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.282456 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:04.282620 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:04.282620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:04.282620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:04.282720 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:04.282771 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.282889 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.282956 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:06:04.283142 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.275930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33249 (local address 127.0.0.1:39950)
0808 21:06:04.276175 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.276186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.276282 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.276448 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.276449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.276792 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:04.276841 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.276845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.277526 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.277536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.278060 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.278062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.278422 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.278423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.278474 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.279054 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:04.279058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:04.280554 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:04.282147 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.282153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.282155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.282342 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.282514 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:04.282514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:04.282515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:04.282567 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:04.282774 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:04.282780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:04.282822 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.282875 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.283024 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":62,"thread_start_us":147,"threads_started":1}
I20260808 21:06:04.285315 6657 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:39950:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 21:06:04.285492 6657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:04.286973 6957 tablet_service.cc:1467] Processing CreateTablet for tablet 995909bd9654417a868fc3fa455b20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c0377e787e349afbcfb40a821c7d0c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:04.286985 6803 tablet_service.cc:1467] Processing CreateTablet for tablet 995909bd9654417a868fc3fa455b20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c0377e787e349afbcfb40a821c7d0c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:04.287200 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995909bd9654417a868fc3fa455b20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.287240 7105 tablet_service.cc:1467] Processing CreateTablet for tablet 995909bd9654417a868fc3fa455b20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c0377e787e349afbcfb40a821c7d0c8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:04.287271 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995909bd9654417a868fc3fa455b20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.287386 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995909bd9654417a868fc3fa455b20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.288753 7202 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: Bootstrap starting.
I20260808 21:06:04.288933 7201 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: Bootstrap starting.
I20260808 21:06:04.289304 7201 tablet_bootstrap.cc:654] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.289394 7202 tablet_bootstrap.cc:654] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.289680 7204 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: Bootstrap starting.
I20260808 21:06:04.289959 7201 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: No bootstrap required, opened a new log
I20260808 21:06:04.290005 7201 ts_tablet_manager.cc:1397] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:04.290174 7201 raft_consensus.cc:348] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.290233 7201 raft_consensus.cc:374] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.290248 7201 raft_consensus.cc:729] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.290303 7201 consensus_queue.cc:260] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.290318 7204 tablet_bootstrap.cc:654] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.290395 7201 ts_tablet_manager.cc:1428] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.290432 7209 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.290508 7209 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.290949 7204 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: No bootstrap required, opened a new log
I20260808 21:06:04.291005 7204 ts_tablet_manager.cc:1397] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:04.291180 7202 tablet_bootstrap.cc:492] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: No bootstrap required, opened a new log
I20260808 21:06:04.291191 7204 raft_consensus.cc:348] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.291234 7202 ts_tablet_manager.cc:1397] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:04.291283 7204 raft_consensus.cc:374] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.291317 7204 raft_consensus.cc:729] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b1463ff52404c88f103d77f11e9f3, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.291366 7204 consensus_queue.cc:260] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.291404 7202 raft_consensus.cc:348] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.291466 7204 ts_tablet_manager.cc:1428] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:04.291476 7202 raft_consensus.cc:374] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.291503 7202 raft_consensus.cc:729] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.291558 7202 consensus_queue.cc:260] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.291638 7202 ts_tablet_manager.cc:1428] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.291599 7210 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.291678 7210 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.291702 7208 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.291786 7208 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260808 21:06:04.337595 6877 tablet.cc:2369] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:04.366986 7208 raft_consensus.cc:482] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:04.367085 7208 raft_consensus.cc:504] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.367578 7208 leader_election.cc:288] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:04.370597 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.367859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:59913 (local address 127.4.177.67:41273)
0808 21:06:04.367905 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.367907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.368099 (+ 192us) server_negotiation.cc:407] Connection header received
0808 21:06:04.368162 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.368163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.368172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.368232 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.368374 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.368911 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.369864 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.370238 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.370317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.370325 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.370490 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.370525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:04.370604 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.367781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33199 (local address 127.4.177.66:40647)
0808 21:06:04.367841 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.367854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.367900 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.368087 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.368089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.368091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.368166 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.368168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.368855 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.368856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.369771 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.369773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.370321 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.370322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.370413 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.370416 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.370486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.370527 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":36}
I20260808 21:06:04.370610 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.367907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:40647 (local address 127.4.177.65:33199)
0808 21:06:04.367950 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.367952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.367961 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:04.367991 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.367992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.368003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.368073 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:04.368254 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.368782 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.369872 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.370236 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.370323 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.370332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.370492 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.370535 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:04.370616 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.367790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:41273 (local address 127.4.177.66:59913)
0808 21:06:04.368042 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.368053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.368106 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.368239 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.368240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.368241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:04.368309 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.368311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.369001 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.369002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.369770 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.369771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.370348 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.370349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.370413 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.370416 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.370482 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.370520 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":42,"thread_start_us":130,"threads_started":1}
I20260808 21:06:04.370893 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995909bd9654417a868fc3fa455b20be" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3" is_pre_election: true
I20260808 21:06:04.370980 6824 raft_consensus.cc:2454] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:04.370970 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995909bd9654417a868fc3fa455b20be" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c" is_pre_election: true
I20260808 21:06:04.371107 7127 raft_consensus.cc:2454] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:04.371203 6906 leader_election.cc:302] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, ae1b1463ff52404c88f103d77f11e9f3; no voters:
I20260808 21:06:04.371363 7208 raft_consensus.cc:2781] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:04.371423 7208 raft_consensus.cc:482] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:04.371455 7208 raft_consensus.cc:3037] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.372144 7208 raft_consensus.cc:504] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.372298 7208 leader_election.cc:288] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Requested vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:04.372447 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995909bd9654417a868fc3fa455b20be" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c"
I20260808 21:06:04.372457 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995909bd9654417a868fc3fa455b20be" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3"
I20260808 21:06:04.372519 7127 raft_consensus.cc:3037] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.372545 6824 raft_consensus.cc:3037] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.373155 6824 raft_consensus.cc:2454] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:04.373167 7127 raft_consensus.cc:2454] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:04.373363 6905 leader_election.cc:302] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, 4f5dc9d0f138468b8d0f742558e0848c; no voters:
I20260808 21:06:04.373466 7208 raft_consensus.cc:2781] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:04.373628 7208 raft_consensus.cc:686] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Running, Role: LEADER
I20260808 21:06:04.373730 7208 consensus_queue.cc:237] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.374159 7224 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: TxnStatusTablet state changed. Reason: New leader 35cb56fabac64cdf969e29a38c15f7a0. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.374261 7224 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:04.374466 7226 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 21:06:04.374588 7226 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 21:06:04.374574 6659 catalog_manager.cc:5495] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:04.386119 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.383729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.4.177.66:37161)
0808 21:06:04.383796 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.383799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.383824 (+ 25us) server_negotiation.cc:407] Connection header received
0808 21:06:04.383911 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.383912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.383931 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.384008 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.384133 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.384580 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.385315 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.385500 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.385581 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.385773 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.385865 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.385949 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.386001 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.386023 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:06:04.386119 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.383674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37161 (local address 127.0.0.1:36918)
0808 21:06:04.383774 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.383786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.383845 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.384016 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.384017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.384018 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:04.384070 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.384073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.384662 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.384664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.385230 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.385232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.385579 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.385580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.385658 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.385680 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.385943 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.385953 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.386000 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.386046 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 21:06:04.388230 6824 raft_consensus.cc:1261] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.388265 7127 raft_consensus.cc:1261] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.388527 7208 consensus_queue.cc:1035] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:04.388724 7224 consensus_queue.cc:1035] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:04.389838 7209 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: TxnStatusTablet state changed. Reason: New leader 35cb56fabac64cdf969e29a38c15f7a0. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.389935 7209 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.390256 7210 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: TxnStatusTablet state changed. Reason: New leader 35cb56fabac64cdf969e29a38c15f7a0. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.390349 7210 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.391217 7224 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.391299 7224 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:04.391501 7208 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.391583 7208 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 35cb56fabac64cdf969e29a38c15f7a0: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:04.391885 7209 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.391971 7209 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P 4f5dc9d0f138468b8d0f742558e0848c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.392055 7210 tablet_replica.cc:439] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } } }
I20260808 21:06:04.392138 7210 tablet_replica.cc:442] T 995909bd9654417a868fc3fa455b20be P ae1b1463ff52404c88f103d77f11e9f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:04.393077 7231 mvcc.cc:205] Tried to move back new op lower bound from 7316370081331445760 to 7316370081275342848. Current Snapshot: MvccSnapshot[applied={T|T < 7316370081331445760 or (T in {7316370081331445760})}]
I20260808 21:06:04.395923 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.393486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.4.177.67:41273)
0808 21:06:04.393540 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.393542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.393551 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:04.393604 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.393605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.393621 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.393688 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.393840 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.394306 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.395081 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.395347 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.395422 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.395596 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.395701 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.395781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.395829 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.395848 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 21:06:04.399495 7230 mvcc.cc:205] Tried to move back new op lower bound from 7316370081331445760 to 7316370081275342848. Current Snapshot: MvccSnapshot[applied={T|T < 7316370081331445760}]
I20260808 21:06:04.401047 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.398481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:36771 (local address 127.4.177.66:37161)
0808 21:06:04.398537 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.398538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.398545 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:06:04.398563 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.398564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.398572 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.398617 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.398769 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.399333 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.400385 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.400562 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.400619 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.400828 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.400852 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.400890 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.400968 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.400983 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 21:06:04.401082 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.398278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37161 (local address 127.4.177.67:36771)
0808 21:06:04.398347 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.398359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.398404 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.398638 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.398639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.398640 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:04.398705 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.398706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.399414 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.399415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.400299 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.400301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.400637 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.400638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.400713 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.400728 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.400910 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.400911 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.400966 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.401010 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 21:06:04.403950 6824 raft_consensus.cc:1261] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Refusing update from remote peer 4f5dc9d0f138468b8d0f742558e0848c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.404031 6977 raft_consensus.cc:1261] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Refusing update from remote peer 4f5dc9d0f138468b8d0f742558e0848c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.404186 7212 consensus_queue.cc:1035] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:04.404487 7212 consensus_queue.cc:1035] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:04.406338 7229 mvcc.cc:205] Tried to move back new op lower bound from 7316370081396756480 to 7316370080698150912. Current Snapshot: MvccSnapshot[applied={T|T < 7316370081396756480}]
I20260808 21:06:04.413059 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.410307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:41877 (local address 127.4.177.67:41273)
0808 21:06:04.410346 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.410348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.410380 (+ 32us) server_negotiation.cc:407] Connection header received
0808 21:06:04.410439 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.410439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.410449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.410506 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.410596 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.411352 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.412452 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.412635 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.412693 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.412877 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.412901 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.412940 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.412992 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.413007 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:04.413101 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.410261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:41273 (local address 127.4.177.66:41877)
0808 21:06:04.410340 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.410349 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.410388 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.410507 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.410507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.410508 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:04.410560 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.410561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.411444 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.411446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.412366 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.412369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.412715 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.412716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.412784 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.412799 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.412941 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.412942 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.412990 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.413026 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 21:06:04.421705 6659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:04.421831 6659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:04.422598 6659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Sending DeleteTablet for 3 replicas of tablet b72cf6818ae94aacb98a285292b7d404
I20260808 21:06:04.422937 6803 tablet_service.cc:1514] Processing DeleteTablet for tablet b72cf6818ae94aacb98a285292b7d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:04 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:45996
I20260808 21:06:04.422971 7105 tablet_service.cc:1514] Processing DeleteTablet for tablet b72cf6818ae94aacb98a285292b7d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:04 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47150
I20260808 21:06:04.423337 7248 tablet_replica.cc:330] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: stopping tablet replica
I20260808 21:06:04.423326 6957 tablet_service.cc:1514] Processing DeleteTablet for tablet b72cf6818ae94aacb98a285292b7d404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:04 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36908
I20260808 21:06:04.423604 7249 tablet_replica.cc:330] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: stopping tablet replica
I20260808 21:06:04.423699 7249 raft_consensus.cc:2229] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:04.423807 7249 raft_consensus.cc:2258] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:04.423832 6659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:04.423978 6659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:04.424265 7249 ts_tablet_manager.cc:1905] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:04.423514 7248 raft_consensus.cc:2229] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:04.424672 7248 raft_consensus.cc:2258] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:04.424911 7250 tablet_replica.cc:330] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: stopping tablet replica
I20260808 21:06:04.425071 7250 raft_consensus.cc:2229] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:04.425179 7250 raft_consensus.cc:2258] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:04.425235 7248 ts_tablet_manager.cc:1905] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:04.425505 6803 tablet_service.cc:1467] Processing CreateTablet for tablet 6bda263ce705499b8336bb8c71b4760f (DEFAULT_TABLE table=test-table [id=cfe8fef4596f4604bb781c293dcce6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.425509 7105 tablet_service.cc:1467] Processing CreateTablet for tablet 6bda263ce705499b8336bb8c71b4760f (DEFAULT_TABLE table=test-table [id=cfe8fef4596f4604bb781c293dcce6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.425623 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bda263ce705499b8336bb8c71b4760f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.425624 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bda263ce705499b8336bb8c71b4760f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.426947 7204 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Bootstrap starting.
I20260808 21:06:04.426994 7201 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Bootstrap starting.
I20260808 21:06:04.427276 6957 tablet_service.cc:1467] Processing CreateTablet for tablet 6bda263ce705499b8336bb8c71b4760f (DEFAULT_TABLE table=test-table [id=cfe8fef4596f4604bb781c293dcce6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:04.427378 7204 tablet_bootstrap.cc:654] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.427392 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bda263ce705499b8336bb8c71b4760f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:04.427452 7201 tablet_bootstrap.cc:654] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.427865 7248 ts_tablet_manager.cc:1918] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:04.427935 7248 log.cc:1196] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/b72cf6818ae94aacb98a285292b7d404
I20260808 21:06:04.428197 7248 ts_tablet_manager.cc:1939] T b72cf6818ae94aacb98a285292b7d404 P ae1b1463ff52404c88f103d77f11e9f3: Deleting consensus metadata
I20260808 21:06:04.428640 7201 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: No bootstrap required, opened a new log
I20260808 21:06:04.428617 6642 catalog_manager.cc:4841] TS ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199): tablet b72cf6818ae94aacb98a285292b7d404 (table test-table [id=0902617e80be4bd690a996edbe7436a0]) successfully deleted
I20260808 21:06:04.428690 7201 ts_tablet_manager.cc:1397] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:04.428848 7201 raft_consensus.cc:348] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.428925 7201 raft_consensus.cc:374] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.428949 7201 raft_consensus.cc:729] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.429005 7202 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Bootstrap starting.
I20260808 21:06:04.428992 7201 consensus_queue.cc:260] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.429106 7201 ts_tablet_manager.cc:1428] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.429425 7202 tablet_bootstrap.cc:654] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:04.430332 7202 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: No bootstrap required, opened a new log
I20260808 21:06:04.430401 7202 ts_tablet_manager.cc:1397] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:04.430542 7202 raft_consensus.cc:348] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.430649 7202 raft_consensus.cc:374] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.430678 7202 raft_consensus.cc:729] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.430721 7202 consensus_queue.cc:260] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.430809 7202 ts_tablet_manager.cc:1428] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:04.432641 7249 ts_tablet_manager.cc:1918] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:04.432726 7249 log.cc:1196] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/b72cf6818ae94aacb98a285292b7d404
I20260808 21:06:04.432986 7249 ts_tablet_manager.cc:1939] T b72cf6818ae94aacb98a285292b7d404 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting consensus metadata
I20260808 21:06:04.433383 6641 catalog_manager.cc:4841] TS 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273): tablet b72cf6818ae94aacb98a285292b7d404 (table test-table [id=0902617e80be4bd690a996edbe7436a0]) successfully deleted
I20260808 21:06:04.435405 7250 ts_tablet_manager.cc:1905] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:04.435547 7204 tablet_bootstrap.cc:492] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: No bootstrap required, opened a new log
I20260808 21:06:04.435606 7204 ts_tablet_manager.cc:1397] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260808 21:06:04.435791 7204 raft_consensus.cc:348] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.435873 7204 raft_consensus.cc:374] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:04.435904 7204 raft_consensus.cc:729] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b1463ff52404c88f103d77f11e9f3, State: Initialized, Role: FOLLOWER
I20260808 21:06:04.435946 7204 consensus_queue.cc:260] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.436053 7204 ts_tablet_manager.cc:1428] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:04.436986 7250 ts_tablet_manager.cc:1918] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:04.437076 7250 log.cc:1196] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/b72cf6818ae94aacb98a285292b7d404
I20260808 21:06:04.437343 7250 ts_tablet_manager.cc:1939] T b72cf6818ae94aacb98a285292b7d404 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting consensus metadata
I20260808 21:06:04.437772 6644 catalog_manager.cc:4841] TS 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161): tablet b72cf6818ae94aacb98a285292b7d404 (table test-table [id=0902617e80be4bd690a996edbe7436a0]) successfully deleted
W20260808 21:06:04.659261 7179 tablet.cc:2369] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:04.746727 7208 raft_consensus.cc:482] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:04.746845 7208 raft_consensus.cc:504] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.747037 7208 leader_election.cc:288] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:04.747368 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bda263ce705499b8336bb8c71b4760f" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c" is_pre_election: true
I20260808 21:06:04.747414 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bda263ce705499b8336bb8c71b4760f" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3" is_pre_election: true
I20260808 21:06:04.747496 7127 raft_consensus.cc:2454] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:04.747548 6824 raft_consensus.cc:2454] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:04.747759 6906 leader_election.cc:302] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, ae1b1463ff52404c88f103d77f11e9f3; no voters:
I20260808 21:06:04.747900 7208 raft_consensus.cc:2781] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:04.747951 7208 raft_consensus.cc:482] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:04.747982 7208 raft_consensus.cc:3037] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.748679 7208 raft_consensus.cc:504] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.748824 7208 leader_election.cc:288] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Requested vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:04.749001 7127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bda263ce705499b8336bb8c71b4760f" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c"
I20260808 21:06:04.749053 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bda263ce705499b8336bb8c71b4760f" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3"
I20260808 21:06:04.749092 7127 raft_consensus.cc:3037] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.749133 6824 raft_consensus.cc:3037] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:04.749765 7127 raft_consensus.cc:2454] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:04.749827 6824 raft_consensus.cc:2454] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:04.749965 6905 leader_election.cc:302] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, 4f5dc9d0f138468b8d0f742558e0848c; no voters:
I20260808 21:06:04.750054 7208 raft_consensus.cc:2781] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:04.750108 7208 raft_consensus.cc:686] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Running, Role: LEADER
I20260808 21:06:04.750165 7208 consensus_queue.cc:237] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:04.750828 6658 catalog_manager.cc:5495] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:04.789881 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.787409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.4.177.66:37161)
0808 21:06:04.787454 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.787456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.787562 (+ 106us) server_negotiation.cc:407] Connection header received
0808 21:06:04.787589 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.787589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.787602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.787648 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.787815 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.788373 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.789196 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.789419 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.789474 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.789628 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.789665 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.789704 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.789746 (+ 42us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.789761 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:04.794519 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.792333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37161 (local address 127.4.177.66:56581)
0808 21:06:04.792430 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:04.792441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:04.792492 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:04.792613 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:04.792613 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:04.792614 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:04.792650 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.792651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.793247 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.793248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.793782 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:04.793784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.794138 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.794138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:04.794192 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.794195 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.794358 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.794360 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:04.794413 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 21:06:04.794448 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 21:06:04.794519 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:04.792376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:56581 (local address 127.4.177.66:37161)
0808 21:06:04.792432 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:04.792434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:04.792490 (+ 56us) server_negotiation.cc:407] Connection header received
0808 21:06:04.792547 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:04.792548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:04.792556 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:04.792610 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:04.792736 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.793164 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.793877 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:04.794056 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:04.794139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:04.794263 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:04.794298 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:04.794359 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:04.794417 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 21:06:04.794431 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 21:06:04.796696 7127 raft_consensus.cc:1261] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.796717 6824 raft_consensus.cc:1261] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:04.796923 7224 consensus_queue.cc:1035] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:04.797032 7208 consensus_queue.cc:1035] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 21:06:04.801435 7229 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 21:06:06.319919 6659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:06.320032 6659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:06.320847 6659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Sending DeleteTablet for 3 replicas of tablet 6bda263ce705499b8336bb8c71b4760f
I20260808 21:06:06.321125 6803 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bda263ce705499b8336bb8c71b4760f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:45996
I20260808 21:06:06.321174 6957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bda263ce705499b8336bb8c71b4760f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36908
I20260808 21:06:06.321376 7281 tablet_replica.cc:330] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: stopping tablet replica
I20260808 21:06:06.321363 7280 tablet_replica.cc:330] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: stopping tablet replica
I20260808 21:06:06.321429 7280 raft_consensus.cc:2229] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:06.321484 7281 raft_consensus.cc:2229] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:06.321519 7280 raft_consensus.cc:2258] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.321730 7281 raft_consensus.cc:2258] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.321801 7280 ts_tablet_manager.cc:1905] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:06.323608 7280 ts_tablet_manager.cc:1918] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 21:06:06.323684 7280 log.cc:1196] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/6bda263ce705499b8336bb8c71b4760f
I20260808 21:06:06.323948 7280 ts_tablet_manager.cc:1939] T 6bda263ce705499b8336bb8c71b4760f P ae1b1463ff52404c88f103d77f11e9f3: Deleting consensus metadata
I20260808 21:06:06.324326 6642 catalog_manager.cc:4841] TS ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199): tablet 6bda263ce705499b8336bb8c71b4760f (table test-table [id=cfe8fef4596f4604bb781c293dcce6f0]) successfully deleted
I20260808 21:06:06.324488 7281 ts_tablet_manager.cc:1905] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:06.324539 7105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bda263ce705499b8336bb8c71b4760f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47150
I20260808 21:06:06.324740 7283 tablet_replica.cc:330] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: stopping tablet replica
I20260808 21:06:06.324803 7283 raft_consensus.cc:2229] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:06.324842 7283 raft_consensus.cc:2258] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.325088 7283 ts_tablet_manager.cc:1905] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:06.326339 7281 ts_tablet_manager.cc:1918] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 21:06:06.326404 7281 log.cc:1196] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/6bda263ce705499b8336bb8c71b4760f
I20260808 21:06:06.326663 7281 ts_tablet_manager.cc:1939] T 6bda263ce705499b8336bb8c71b4760f P 35cb56fabac64cdf969e29a38c15f7a0: Deleting consensus metadata
I20260808 21:06:06.326995 7283 ts_tablet_manager.cc:1918] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 21:06:06.327003 6644 catalog_manager.cc:4841] TS 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161): tablet 6bda263ce705499b8336bb8c71b4760f (table test-table [id=cfe8fef4596f4604bb781c293dcce6f0]) successfully deleted
I20260808 21:06:06.327067 7283 log.cc:1196] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/6bda263ce705499b8336bb8c71b4760f
I20260808 21:06:06.327308 7283 ts_tablet_manager.cc:1939] T 6bda263ce705499b8336bb8c71b4760f P 4f5dc9d0f138468b8d0f742558e0848c: Deleting consensus metadata
I20260808 21:06:06.327683 6641 catalog_manager.cc:4841] TS 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273): tablet 6bda263ce705499b8336bb8c71b4760f (table test-table [id=cfe8fef4596f4604bb781c293dcce6f0]) successfully deleted
Aug 08 21:06:06 dist-test-slave-chsk krb5kdc[6611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 21:06:06 dist-test-slave-chsk krb5kdc[6611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223166, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:06.346156 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:06.338951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.4.177.126:33249)
0808 21:06:06.339087 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:06.339089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:06.339100 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:06.339283 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:06.339284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:06.339296 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:06.339364 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:06.339961 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:06.340585 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:06.341323 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:06.341516 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:06.341568 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:06.343971 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:06.343985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:06.343986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:06.343993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:06.345275 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:06.345564 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:06.345565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:06.345565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:06.345616 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:06.345778 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:06.345778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:06.345779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:06.345875 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:06.345947 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:06.346032 (+ 85us) server_negotiation.cc:299] Negotiation successful
0808 21:06:06.346090 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260808 21:06:06.347661 6659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:06.347820 6659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:06.349362 7105 tablet_service.cc:1467] Processing CreateTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 (DEFAULT_TABLE table=test-table [id=097e199d148649f8a475e9ad8a13a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.349398 6957 tablet_service.cc:1467] Processing CreateTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 (DEFAULT_TABLE table=test-table [id=097e199d148649f8a475e9ad8a13a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.349519 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e23a6560dab42ab9cd286df01a4c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.349531 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e23a6560dab42ab9cd286df01a4c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.350602 7292 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Bootstrap starting.
I20260808 21:06:06.350612 7291 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Bootstrap starting.
I20260808 21:06:06.350642 6803 tablet_service.cc:1467] Processing CreateTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 (DEFAULT_TABLE table=test-table [id=097e199d148649f8a475e9ad8a13a788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.350762 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e23a6560dab42ab9cd286df01a4c5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.350975 7292 tablet_bootstrap.cc:654] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.351016 7291 tablet_bootstrap.cc:654] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.351657 7291 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: No bootstrap required, opened a new log
I20260808 21:06:06.351719 7291 ts_tablet_manager.cc:1397] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:06.351831 7291 raft_consensus.cc:348] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.351905 7291 raft_consensus.cc:374] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.351925 7291 raft_consensus.cc:729] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.351994 7291 consensus_queue.cc:260] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.352203 7291 ts_tablet_manager.cc:1428] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:06.352237 7298 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Bootstrap starting.
I20260808 21:06:06.352645 7298 tablet_bootstrap.cc:654] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.352689 7292 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: No bootstrap required, opened a new log
I20260808 21:06:06.352736 7292 ts_tablet_manager.cc:1397] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:06.352857 7292 raft_consensus.cc:348] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.352939 7292 raft_consensus.cc:374] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.352970 7292 raft_consensus.cc:729] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.353010 7292 consensus_queue.cc:260] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.353128 7292 ts_tablet_manager.cc:1428] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:06.353396 7298 tablet_bootstrap.cc:492] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: No bootstrap required, opened a new log
I20260808 21:06:06.353456 7298 ts_tablet_manager.cc:1397] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:06.353565 7298 raft_consensus.cc:348] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.353642 7298 raft_consensus.cc:374] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.353670 7298 raft_consensus.cc:729] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b1463ff52404c88f103d77f11e9f3, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.353717 7298 consensus_queue.cc:260] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.353895 7298 ts_tablet_manager.cc:1428] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:06.681119 7262 raft_consensus.cc:482] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:06.681231 7262 raft_consensus.cc:504] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.681464 7262 leader_election.cc:288] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:06.681756 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e23a6560dab42ab9cd286df01a4c5a4" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c" is_pre_election: true
I20260808 21:06:06.681761 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e23a6560dab42ab9cd286df01a4c5a4" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3" is_pre_election: true
I20260808 21:06:06.681877 6824 raft_consensus.cc:2454] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:06.681877 7126 raft_consensus.cc:2454] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:06.682071 6905 leader_election.cc:302] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, 4f5dc9d0f138468b8d0f742558e0848c; no voters:
I20260808 21:06:06.682188 7262 raft_consensus.cc:2781] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:06.682247 7262 raft_consensus.cc:482] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:06.682265 7262 raft_consensus.cc:3037] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.682945 7262 raft_consensus.cc:504] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.683087 7262 leader_election.cc:288] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Requested vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:06.683248 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e23a6560dab42ab9cd286df01a4c5a4" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3"
I20260808 21:06:06.683279 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e23a6560dab42ab9cd286df01a4c5a4" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c"
I20260808 21:06:06.683354 6824 raft_consensus.cc:3037] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.683357 7126 raft_consensus.cc:3037] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.683976 6824 raft_consensus.cc:2454] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:06.684020 7126 raft_consensus.cc:2454] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:06.684187 6906 leader_election.cc:302] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, ae1b1463ff52404c88f103d77f11e9f3; no voters:
I20260808 21:06:06.684312 7262 raft_consensus.cc:2781] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:06.684361 7262 raft_consensus.cc:686] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Running, Role: LEADER
I20260808 21:06:06.684415 7262 consensus_queue.cc:237] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.685088 6658 catalog_manager.cc:5495] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:06.709860 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:06.707385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.4.177.66:37161)
0808 21:06:06.707551 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:06.707554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:06.707594 (+ 40us) server_negotiation.cc:407] Connection header received
0808 21:06:06.707671 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:06.707672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:06.707685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:06.707761 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:06.707898 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:06.708388 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:06.709169 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:06.709386 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:06.709462 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:06.709616 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:06.709650 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:06.709699 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:06.709747 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 21:06:06.709769 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":115,"threads_started":1}
I20260808 21:06:06.712019 6824 raft_consensus.cc:1261] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:06.712163 7126 raft_consensus.cc:1261] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:06.712226 7241 consensus_queue.cc:1035] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:06.712352 7262 consensus_queue.cc:1035] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:06.721467 6659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:06.721577 6659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:06.722180 6659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Sending DeleteTablet for 3 replicas of tablet 6e23a6560dab42ab9cd286df01a4c5a4
I20260808 21:06:06.722442 7105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47150
I20260808 21:06:06.722467 6803 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:45996
I20260808 21:06:06.722556 7283 tablet_replica.cc:330] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: stopping tablet replica
I20260808 21:06:06.722564 7280 tablet_replica.cc:330] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: stopping tablet replica
I20260808 21:06:06.722610 7280 raft_consensus.cc:2229] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:06.722612 7283 raft_consensus.cc:2229] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:06.722651 7280 raft_consensus.cc:2258] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.722658 7283 raft_consensus.cc:2258] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.722741 6957 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e23a6560dab42ab9cd286df01a4c5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:06 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36908
I20260808 21:06:06.722831 7281 tablet_replica.cc:330] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: stopping tablet replica
I20260808 21:06:06.722888 7281 raft_consensus.cc:2229] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:06.722949 7281 raft_consensus.cc:2258] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:06.722939 6659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260808 21:06:06.723004 7280 ts_tablet_manager.cc:1905] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 21:06:06.723081 6659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:06.723286 7281 ts_tablet_manager.cc:1905] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:06.724227 6803 tablet_service.cc:1467] Processing CreateTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb (DEFAULT_TABLE table=test-table [id=e67c452c5d604edd98f57b50bdb41ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.724334 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e84c4361634422c8b3a260d8b33f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.724385 7105 tablet_service.cc:1467] Processing CreateTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb (DEFAULT_TABLE table=test-table [id=e67c452c5d604edd98f57b50bdb41ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.724489 7105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e84c4361634422c8b3a260d8b33f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.724629 7280 ts_tablet_manager.cc:1918] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:06.724675 7280 log.cc:1196] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/6e23a6560dab42ab9cd286df01a4c5a4
I20260808 21:06:06.724908 7280 ts_tablet_manager.cc:1939] T 6e23a6560dab42ab9cd286df01a4c5a4 P ae1b1463ff52404c88f103d77f11e9f3: Deleting consensus metadata
I20260808 21:06:06.725003 7281 ts_tablet_manager.cc:1918] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:06.725054 7281 log.cc:1196] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/6e23a6560dab42ab9cd286df01a4c5a4
I20260808 21:06:06.725271 7281 ts_tablet_manager.cc:1939] T 6e23a6560dab42ab9cd286df01a4c5a4 P 35cb56fabac64cdf969e29a38c15f7a0: Deleting consensus metadata
I20260808 21:06:06.725571 7298 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3: Bootstrap starting.
I20260808 21:06:06.725567 7283 ts_tablet_manager.cc:1905] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:06.725852 7291 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Bootstrap starting.
I20260808 21:06:06.725999 7298 tablet_bootstrap.cc:654] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.726166 7291 tablet_bootstrap.cc:654] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.726572 6644 catalog_manager.cc:4841] TS 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161): tablet 6e23a6560dab42ab9cd286df01a4c5a4 (table test-table [id=097e199d148649f8a475e9ad8a13a788]) successfully deleted
I20260808 21:06:06.726622 7298 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3: No bootstrap required, opened a new log
I20260808 21:06:06.726678 6957 tablet_service.cc:1467] Processing CreateTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb (DEFAULT_TABLE table=test-table [id=e67c452c5d604edd98f57b50bdb41ecf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:06.726697 7298 ts_tablet_manager.cc:1397] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:06.726788 6957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e84c4361634422c8b3a260d8b33f6eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:06.726847 7298 raft_consensus.cc:348] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.726939 7298 raft_consensus.cc:374] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.726966 7298 raft_consensus.cc:729] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae1b1463ff52404c88f103d77f11e9f3, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.726980 7283 ts_tablet_manager.cc:1918] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:06.727036 7283 log.cc:1196] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/6e23a6560dab42ab9cd286df01a4c5a4
I20260808 21:06:06.727013 7298 consensus_queue.cc:260] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.727106 7298 ts_tablet_manager.cc:1428] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:06.727269 7283 ts_tablet_manager.cc:1939] T 6e23a6560dab42ab9cd286df01a4c5a4 P 4f5dc9d0f138468b8d0f742558e0848c: Deleting consensus metadata
I20260808 21:06:06.727352 6642 catalog_manager.cc:4841] TS ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199): tablet 6e23a6560dab42ab9cd286df01a4c5a4 (table test-table [id=097e199d148649f8a475e9ad8a13a788]) successfully deleted
I20260808 21:06:06.727895 6641 catalog_manager.cc:4841] TS 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273): tablet 6e23a6560dab42ab9cd286df01a4c5a4 (table test-table [id=097e199d148649f8a475e9ad8a13a788]) successfully deleted
I20260808 21:06:06.728184 7292 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Bootstrap starting.
I20260808 21:06:06.728475 7292 tablet_bootstrap.cc:654] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:06.728624 7291 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: No bootstrap required, opened a new log
I20260808 21:06:06.728679 7291 ts_tablet_manager.cc:1397] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260808 21:06:06.728780 7291 raft_consensus.cc:348] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.728829 7291 raft_consensus.cc:374] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.728842 7291 raft_consensus.cc:729] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5dc9d0f138468b8d0f742558e0848c, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.728896 7291 consensus_queue.cc:260] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.728991 7291 ts_tablet_manager.cc:1428] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:06.729125 7292 tablet_bootstrap.cc:492] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: No bootstrap required, opened a new log
I20260808 21:06:06.729197 7292 ts_tablet_manager.cc:1397] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:06.729300 7292 raft_consensus.cc:348] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.729383 7292 raft_consensus.cc:374] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:06.729408 7292 raft_consensus.cc:729] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Initialized, Role: FOLLOWER
I20260808 21:06:06.729454 7292 consensus_queue.cc:260] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.729550 7292 ts_tablet_manager.cc:1428] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:06.873147 7267 raft_consensus.cc:482] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:06.873265 7267 raft_consensus.cc:504] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.873479 7267 leader_election.cc:288] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:06.873680 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e84c4361634422c8b3a260d8b33f6eb" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c" is_pre_election: true
I20260808 21:06:06.873721 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e84c4361634422c8b3a260d8b33f6eb" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3" is_pre_election: true
I20260808 21:06:06.873816 6824 raft_consensus.cc:2454] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:06.873818 7126 raft_consensus.cc:2454] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 0.
I20260808 21:06:06.874002 6906 leader_election.cc:302] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, ae1b1463ff52404c88f103d77f11e9f3; no voters:
I20260808 21:06:06.874118 7267 raft_consensus.cc:2781] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:06.874161 7267 raft_consensus.cc:482] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:06.874186 7267 raft_consensus.cc:3037] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.874867 7267 raft_consensus.cc:504] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.875003 7267 leader_election.cc:288] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Requested vote from peers ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199), 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273)
I20260808 21:06:06.875190 6824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e84c4361634422c8b3a260d8b33f6eb" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae1b1463ff52404c88f103d77f11e9f3"
I20260808 21:06:06.875202 7126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e84c4361634422c8b3a260d8b33f6eb" candidate_uuid: "35cb56fabac64cdf969e29a38c15f7a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5dc9d0f138468b8d0f742558e0848c"
I20260808 21:06:06.875280 7126 raft_consensus.cc:3037] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.875284 6824 raft_consensus.cc:3037] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:06.875947 7126 raft_consensus.cc:2454] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:06.876034 6824 raft_consensus.cc:2454] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cb56fabac64cdf969e29a38c15f7a0 in term 1.
I20260808 21:06:06.876135 6905 leader_election.cc:302] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cb56fabac64cdf969e29a38c15f7a0, 4f5dc9d0f138468b8d0f742558e0848c; no voters:
I20260808 21:06:06.876262 7267 raft_consensus.cc:2781] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:06.876350 7267 raft_consensus.cc:686] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35cb56fabac64cdf969e29a38c15f7a0, State: Running, Role: LEADER
I20260808 21:06:06.876426 7267 consensus_queue.cc:237] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } }
I20260808 21:06:06.877053 6659 catalog_manager.cc:5495] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "35cb56fabac64cdf969e29a38c15f7a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cb56fabac64cdf969e29a38c15f7a0" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:06.912375 7126 raft_consensus.cc:1261] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:06.912375 6824 raft_consensus.cc:1261] T 9e84c4361634422c8b3a260d8b33f6eb P ae1b1463ff52404c88f103d77f11e9f3 [term 1 FOLLOWER]: Refusing update from remote peer 35cb56fabac64cdf969e29a38c15f7a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:06.912673 7267 consensus_queue.cc:1035] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5dc9d0f138468b8d0f742558e0848c" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:06.912815 7262 consensus_queue.cc:1035] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae1b1463ff52404c88f103d77f11e9f3" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 21:06:06.917129 7259 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 21:06:08.434342 6658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:08.434446 6658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:08.435324 6658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8da7b948306a41bd80ba30767fda906e: Sending DeleteTablet for 3 replicas of tablet 9e84c4361634422c8b3a260d8b33f6eb
I20260808 21:06:08.435570 6803 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:08 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:45996
I20260808 21:06:08.435581 6957 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:08 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36908
I20260808 21:06:08.435604 7105 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e84c4361634422c8b3a260d8b33f6eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:08 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47150
I20260808 21:06:08.435732 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6735
I20260808 21:06:08.440402 7337 tablet_replica.cc:330] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: stopping tablet replica
I20260808 21:06:08.440493 7337 raft_consensus.cc:2229] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:08.440568 7337 raft_consensus.cc:2258] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:08.440687 7336 tablet_replica.cc:330] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: stopping tablet replica
I20260808 21:06:08.440755 7336 raft_consensus.cc:2229] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:08.440824 7336 raft_consensus.cc:2258] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:08.440843 7337 ts_tablet_manager.cc:1905] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:08.441140 7336 ts_tablet_manager.cc:1905] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:08.442720 7336 ts_tablet_manager.cc:1918] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 21:06:08.442798 7336 log.cc:1196] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/9e84c4361634422c8b3a260d8b33f6eb
I20260808 21:06:08.443096 7336 ts_tablet_manager.cc:1939] T 9e84c4361634422c8b3a260d8b33f6eb P 4f5dc9d0f138468b8d0f742558e0848c: Deleting consensus metadata
I20260808 21:06:08.443181 7337 ts_tablet_manager.cc:1918] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 21:06:08.443254 7337 log.cc:1196] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/9e84c4361634422c8b3a260d8b33f6eb
I20260808 21:06:08.443552 7337 ts_tablet_manager.cc:1939] T 9e84c4361634422c8b3a260d8b33f6eb P 35cb56fabac64cdf969e29a38c15f7a0: Deleting consensus metadata
I20260808 21:06:08.443535 6641 catalog_manager.cc:4841] TS 4f5dc9d0f138468b8d0f742558e0848c (127.4.177.67:41273): tablet 9e84c4361634422c8b3a260d8b33f6eb (table test-table [id=e67c452c5d604edd98f57b50bdb41ecf]) successfully deleted
I20260808 21:06:08.443959 6644 catalog_manager.cc:4841] TS 35cb56fabac64cdf969e29a38c15f7a0 (127.4.177.66:37161): tablet 9e84c4361634422c8b3a260d8b33f6eb (table test-table [id=e67c452c5d604edd98f57b50bdb41ecf]) successfully deleted
W20260808 21:06:08.444784 6642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:33199 (error 108)
I20260808 21:06:08.445384 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:08.445179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33199 (local address 127.0.0.1:56112)
0808 21:06:08.445307 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:08.445349 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260808 21:06:08.445498 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6887
W20260808 21:06:08.445528 6642 catalog_manager.cc:4584] TS ae1b1463ff52404c88f103d77f11e9f3 (127.4.177.65:33199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e84c4361634422c8b3a260d8b33f6eb: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33199: connect: Connection refused (error 111)
I20260808 21:06:08.456607 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7037
I20260808 21:06:08.464180 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 6624
2026-08-08T21:06:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6138 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:08 dist-test-slave-chsk krb5kdc[7343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:08 dist-test-slave-chsk krb5kdc[7343](info): set up 2 sockets
Aug 08 21:06:08 dist-test-slave-chsk krb5kdc[7343](info): commencing operation
krb5kdc: starting...
W20260808 21:06:09.539944 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:09 dist-test-slave-chsk krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:09.662621 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43993
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:09.744451 7357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:09.744675 7357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:09.744714 7357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:09.745837 7357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:09.745901 7357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:09.745929 7357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:09.745962 7357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:09.745985 7357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:09.747493 7357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:09.747758 7357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:09.747974 7357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:09.752765 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:09.752826 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:09.752939 7357 server_base.cc:1034] running on GCE node
W20260808 21:06:09.752861 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:09.753245 7357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:09.753480 7357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:09.754621 7357 hybrid_clock.cc:648] HybridClock initialized: now 1786223169754599 us; error 39 us; skew 500 ppm
Aug 08 21:06:09 dist-test-slave-chsk krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:09.757099 7357 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:09.757579 7357 webserver.cc:460] Webserver started at http://127.4.177.126:37181/ using document root <none> and password file <none>
I20260808 21:06:09.757781 7357 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:09.757822 7357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:09.757907 7357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:09.758845 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "0e1b45358d11489fbda6adf28de82ac1"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "0d4a5ae6078fff9bdbdb748c901d1171"
server_key_iv: "4b3252cc3b220e7a2a22dac7dad6ca61"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.759132 7357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "0e1b45358d11489fbda6adf28de82ac1"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "0d4a5ae6078fff9bdbdb748c901d1171"
server_key_iv: "4b3252cc3b220e7a2a22dac7dad6ca61"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.760454 7357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:09.761133 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:09.761327 7357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:09.761420 7357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "0e1b45358d11489fbda6adf28de82ac1"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "0d4a5ae6078fff9bdbdb748c901d1171"
server_key_iv: "4b3252cc3b220e7a2a22dac7dad6ca61"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.761493 7357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:09.772920 7357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:09.775111 7357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:09.775322 7357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:09.779613 7357 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:43993
I20260808 21:06:09.779665 7428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:43993 every 8 connection(s)
I20260808 21:06:09.779994 7357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:09.780879 7429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:09.783121 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: Bootstrap starting.
I20260808 21:06:09.783788 7429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:09.784096 7429 log.cc:824] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:09.784782 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: No bootstrap required, opened a new log
I20260808 21:06:09.786448 7429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } }
I20260808 21:06:09.786584 7429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:09.786643 7429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1b45358d11489fbda6adf28de82ac1, State: Initialized, Role: FOLLOWER
I20260808 21:06:09.786785 7429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } }
I20260808 21:06:09.786849 7429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:09.786908 7429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:09.786964 7429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:09.787559 7429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } }
I20260808 21:06:09.787678 7429 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e1b45358d11489fbda6adf28de82ac1; no voters:
I20260808 21:06:09.787882 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7357
I20260808 21:06:09.787899 7429 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:09.787930 7432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:09.787973 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:09.788143 4805 external_mini_cluster.cc:1428] Setting key 276070cc2da5d5b1f1f15ea6ba373b5b
I20260808 21:06:09.788162 7429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:09.788475 7432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1b45358d11489fbda6adf28de82ac1, State: Running, Role: LEADER
I20260808 21:06:09.788560 7432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } }
I20260808 21:06:09.788916 7432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1b45358d11489fbda6adf28de82ac1. Latest consensus state: current_term: 1 leader_uuid: "0e1b45358d11489fbda6adf28de82ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } } }
I20260808 21:06:09.788929 7433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e1b45358d11489fbda6adf28de82ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e1b45358d11489fbda6adf28de82ac1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43993 } } }
I20260808 21:06:09.789117 7433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:09.789337 7432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1 [sys.catalog]: This master's current role is: LEADER
W20260808 21:06:09.790410 7452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:09.790468 7452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:09.790522 7444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:09.790673 7444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:09.792961 7444 catalog_manager.cc:1347] Generated new cluster ID: 4b51a7e2b01847f695ec9836fb70e345
I20260808 21:06:09.793017 7444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:09 dist-test-slave-chsk krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:09.799564 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:09.788571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.4.177.126:43993)
0808 21:06:09.788813 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:09.788824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:09.789035 (+ 211us) server_negotiation.cc:407] Connection header received
0808 21:06:09.789157 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:09.789164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:09.789220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:09.789355 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:09.790325 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.791532 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.792580 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.793067 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.793185 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:09.795558 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:09.795562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:09.795562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:09.795573 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:09.797863 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:09.798229 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:09.798232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:09.798233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:09.798321 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:09.798533 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:09.798534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:09.798534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:09.798841 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:09.799061 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:09.799135 (+ 74us) server_negotiation.cc:299] Negotiation successful
0808 21:06:09.799406 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260808 21:06:09.814805 7444 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:09.815564 7444 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:09.823553 7444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e1b45358d11489fbda6adf28de82ac1: Generated new TSK 0
I20260808 21:06:09.823740 7444 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:09.857465 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:09.939857 7457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:09.940066 7457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:09.940090 7457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:09.941179 7457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:09.941213 7457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:09.941241 7457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:09.942698 7457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:09.942907 7457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:09.943092 7457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:09.948053 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:09.948189 7457 server_base.cc:1034] running on GCE node
W20260808 21:06:09.948058 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:09.948135 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:09.948583 7457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:09.948807 7457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:09.950011 7457 hybrid_clock.cc:648] HybridClock initialized: now 1786223169949987 us; error 37 us; skew 500 ppm
Aug 08 21:06:09 dist-test-slave-chsk krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:09.952641 7457 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:09.953135 7457 webserver.cc:460] Webserver started at http://127.4.177.65:43439/ using document root <none> and password file <none>
I20260808 21:06:09.953372 7457 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:09.953439 7457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:09.953603 7457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:09.954674 7457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "278fb0068c864943994b3957b7d2c904"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "584e4112a5cd2724bbfd122622bd396c"
server_key_iv: "87ead89450ffe7fe84872ddc9f825ebf"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.955049 7457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "278fb0068c864943994b3957b7d2c904"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "584e4112a5cd2724bbfd122622bd396c"
server_key_iv: "87ead89450ffe7fe84872ddc9f825ebf"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.956442 7457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:09.957173 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:09.957356 7457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:09.957460 7457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "278fb0068c864943994b3957b7d2c904"
format_stamp: "Formatted at 2026-08-08 21:06:09 on dist-test-slave-chsk"
server_key: "584e4112a5cd2724bbfd122622bd396c"
server_key_iv: "87ead89450ffe7fe84872ddc9f825ebf"
server_key_version: "encryptionkey@0"
I20260808 21:06:09.957556 7457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:09.967515 7457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:09.969233 7457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:09.969395 7457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:09.969626 7457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:09.969938 7457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:09.969993 7457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:09.970027 7457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:09.970055 7457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:09.975979 7457 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:40675
I20260808 21:06:09.976052 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:40675 every 8 connection(s)
I20260808 21:06:09.976362 7457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:06:09 dist-test-slave-chsk krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223169, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:09.982697 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7457
I20260808 21:06:09.982801 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:09.983027 4805 external_mini_cluster.cc:1428] Setting key 72646b388fe70d0e91d7380c08971346
I20260808 21:06:09.985891 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:09.977157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:58627 (local address 127.4.177.126:43993)
0808 21:06:09.977228 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:09.977230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:09.977403 (+ 173us) server_negotiation.cc:407] Connection header received
0808 21:06:09.977873 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:09.977874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:09.977887 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:09.977946 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:09.978841 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.979414 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.980351 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.980551 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.980608 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:09.982702 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:09.982703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:09.982704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:09.982713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:09.984304 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:09.984806 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:09.984807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:09.984808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:09.984864 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:09.985147 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:09.985148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:09.985148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:09.985253 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:09.985305 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:09.985661 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 21:06:09.985763 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 21:06:09.985996 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:09.977111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43993 (local address 127.4.177.65:58627)
0808 21:06:09.977272 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:09.977314 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:09.977782 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:09.977994 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:09.977997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:09.978562 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:09.978756 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:09.978760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.979493 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.979493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:09.980274 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:09.980276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:09.980626 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:09.980627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:09.980732 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:09.981246 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:09.981255 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:09.982604 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:09.984390 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:09.984398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:09.984400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:09.984726 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:09.984934 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:09.984934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:09.984935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:09.985081 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:09.985317 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:09.985323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:09.985495 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 21:06:09.985624 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 21:06:09.985789 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260808 21:06:09.986516 7592 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43993
I20260808 21:06:09.986613 7592 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:09.986819 7592 heartbeater.cc:511] Master 127.4.177.126:43993 requested a full tablet report, sending...
I20260808 21:06:09.987375 7390 ts_manager.cc:194] Registered new tserver with Master: 278fb0068c864943994b3957b7d2c904 (127.4.177.65:40675)
I20260808 21:06:09.988006 7390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:58627
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:10.009514 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:10.091607 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:10.091861 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:10.091902 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:10.093078 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:10.093146 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:10.093189 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:10.094705 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:10.094985 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:10.095263 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:10.100302 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:10.100605 7597 server_base.cc:1034] running on GCE node
W20260808 21:06:10.100319 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:10.100343 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:10.101075 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:10.101369 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:10.102521 7597 hybrid_clock.cc:648] HybridClock initialized: now 1786223170102501 us; error 38 us; skew 500 ppm
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:10.105134 7597 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:10.105536 7597 webserver.cc:460] Webserver started at http://127.4.177.66:44529/ using document root <none> and password file <none>
I20260808 21:06:10.105741 7597 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:10.105783 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:10.105873 7597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:10.106895 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "2308fa3ff73b41bbb7fd232a7535e25b"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "af97e2e5ad6a586cad040412d432a377"
server_key_iv: "0085b08ad63f7b843e2b59b818739148"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.107282 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "2308fa3ff73b41bbb7fd232a7535e25b"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "af97e2e5ad6a586cad040412d432a377"
server_key_iv: "0085b08ad63f7b843e2b59b818739148"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.108618 7597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:10.109329 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.109504 7597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:10.109596 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "2308fa3ff73b41bbb7fd232a7535e25b"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "af97e2e5ad6a586cad040412d432a377"
server_key_iv: "0085b08ad63f7b843e2b59b818739148"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.109653 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:10.124053 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:10.125952 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:10.126073 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:10.126271 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:10.126606 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:10.126642 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.126665 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:10.126724 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.132603 7597 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38045
I20260808 21:06:10.132683 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38045 every 8 connection(s)
I20260808 21:06:10.133033 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:10.134588 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7597
I20260808 21:06:10.134704 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:10.134940 4805 external_mini_cluster.cc:1428] Setting key 85bdc8cf87407246872e2e38fe18895d
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:10.144042 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:10.133849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:41727 (local address 127.4.177.126:43993)
0808 21:06:10.133937 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:10.133940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:10.134098 (+ 158us) server_negotiation.cc:407] Connection header received
0808 21:06:10.134765 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:10.134767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:10.134780 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:10.134839 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:10.136208 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.136710 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.138066 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.138373 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.138451 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:10.141043 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:10.141045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:10.141045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:10.141052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:10.142521 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:10.143027 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:10.143028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:10.143029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:10.143080 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:10.143391 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:10.143391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:10.143392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:10.143499 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:10.143550 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:10.143902 (+ 352us) server_negotiation.cc:299] Negotiation successful
0808 21:06:10.143975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 21:06:10.144305 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:10.133794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43993 (local address 127.4.177.66:41727)
0808 21:06:10.133976 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:10.134014 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:10.134596 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:10.134953 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:10.134958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:10.135806 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:10.136095 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:10.136101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.136803 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.136805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:10.137979 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:10.137981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.138460 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.138461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:10.138558 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:10.139099 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:10.139109 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:10.140948 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:10.142608 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:10.142616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:10.142618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:10.142951 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:10.143164 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:10.143165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:10.143166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:10.143313 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:10.143568 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:10.143575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:10.143754 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 21:06:10.143873 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 21:06:10.144055 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260808 21:06:10.144929 7746 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43993
I20260808 21:06:10.145023 7746 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
I20260808 21:06:10.145228 7746 heartbeater.cc:511] Master 127.4.177.126:43993 requested a full tablet report, sending...
I20260808 21:06:10.145649 7390 ts_manager.cc:194] Registered new tserver with Master: 2308fa3ff73b41bbb7fd232a7535e25b (127.4.177.66:38045)
I20260808 21:06:10.146030 7390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:41727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:10.162253 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:10.244681 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:10.244952 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:10.244993 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:10.246165 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:10.246232 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:10.246276 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:10.247751 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:10.248023 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:10.248263 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:10.253250 7758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:10.253262 7757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:10.253265 7760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:10.253496 7751 server_base.cc:1034] running on GCE node
I20260808 21:06:10.253700 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:10.253913 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:10.255121 7751 hybrid_clock.cc:648] HybridClock initialized: now 1786223170255094 us; error 37 us; skew 500 ppm
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:10.257735 7751 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:10.258219 7751 webserver.cc:460] Webserver started at http://127.4.177.67:41711/ using document root <none> and password file <none>
I20260808 21:06:10.258463 7751 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:10.258533 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:10.258690 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:10.259855 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "c7863705999747dbbc5d01d1a8f07cd1"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "f4bf2a49685afe37e27af031455711e1"
server_key_iv: "db7280b84f68c93f7f2999b753bd8e50"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.260242 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "c7863705999747dbbc5d01d1a8f07cd1"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "f4bf2a49685afe37e27af031455711e1"
server_key_iv: "db7280b84f68c93f7f2999b753bd8e50"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.261675 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:06:10.262553 7768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.262753 7751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:10.262863 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "c7863705999747dbbc5d01d1a8f07cd1"
format_stamp: "Formatted at 2026-08-08 21:06:10 on dist-test-slave-chsk"
server_key: "f4bf2a49685afe37e27af031455711e1"
server_key_iv: "db7280b84f68c93f7f2999b753bd8e50"
server_key_version: "encryptionkey@0"
I20260808 21:06:10.262949 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:10.270464 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:10.272436 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:10.272632 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:10.272902 7751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:10.273196 7751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:10.273239 7751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.273263 7751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:10.273277 7751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:10.279456 7751 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:45787
I20260808 21:06:10.279502 7889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:45787 every 8 connection(s)
I20260808 21:06:10.279878 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223170, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:10.287652 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7751
I20260808 21:06:10.287781 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:10.287989 4805 external_mini_cluster.cc:1428] Setting key de9500634270d41dc850da1b6f7d3bcb
I20260808 21:06:10.289855 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:10.280676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:51071 (local address 127.4.177.126:43993)
0808 21:06:10.280765 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:10.280767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:10.280961 (+ 194us) server_negotiation.cc:407] Connection header received
0808 21:06:10.281432 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:10.281433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:10.281445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:10.281496 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:10.282477 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.282967 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.284008 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.284196 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.284260 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:10.286569 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:10.286571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:10.286571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:10.286579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:10.288148 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:10.288739 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:10.288740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:10.288741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:10.288807 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:10.289107 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:10.289108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:10.289109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:10.289302 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:10.289370 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:10.289705 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 21:06:10.289781 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 21:06:10.290107 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:10.280631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43993 (local address 127.4.177.67:51071)
0808 21:06:10.280831 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:10.280869 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:10.281347 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:10.281558 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:10.281561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:10.282189 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:10.282393 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:10.282398 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.283048 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.283048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:10.283927 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:10.283929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:10.284284 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:10.284284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:10.284394 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:10.285003 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:10.285012 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:10.286467 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:10.288255 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:10.288264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:10.288266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:10.288651 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:10.288886 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:10.288886 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:10.288887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:10.289035 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:10.289383 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:10.289391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:10.289568 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 21:06:10.289667 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 21:06:10.289862 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
I20260808 21:06:10.290578 7890 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43993
I20260808 21:06:10.290674 7890 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:10.290904 7890 heartbeater.cc:511] Master 127.4.177.126:43993 requested a full tablet report, sending...
I20260808 21:06:10.291326 7390 ts_manager.cc:194] Registered new tserver with Master: c7863705999747dbbc5d01d1a8f07cd1 (127.4.177.67:45787)
I20260808 21:06:10.291718 7390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:51071
I20260808 21:06:10.300160 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 21:06:10.313158 7436 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:10.312053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.4.177.126:43993)
0808 21:06:10.312137 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:10.312140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:10.312221 (+ 81us) server_negotiation.cc:407] Connection header received
0808 21:06:10.312326 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:10.312327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:10.312338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:10.312436 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:10.313075 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37414: BlockingRecv error: recv got EOF from 127.0.0.1:37414 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:06:10.314030 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7457
I20260808 21:06:10.320266 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7597
I20260808 21:06:10.326359 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7751
I20260808 21:06:10.331696 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7357
2026-08-08T21:06:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1866 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7902](info): set up 2 sockets
Aug 08 21:06:10 dist-test-slave-chsk krb5kdc[7902](info): commencing operation
krb5kdc: starting...
W20260808 21:06:11.408013 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[7902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:11.527760 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38919
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:34875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:38919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:11.610469 7915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:11.610703 7915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:11.610739 7915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:11.611949 7915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:11.612020 7915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:11.612044 7915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:11.612072 7915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:11.612104 7915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:11.613617 7915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:34875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:38919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:11.613895 7915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:11.614122 7915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:11.618960 7921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:11.619006 7915 server_base.cc:1034] running on GCE node
W20260808 21:06:11.618973 7920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:11.618973 7923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:11.619467 7915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:11.619773 7915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:11.620939 7915 hybrid_clock.cc:648] HybridClock initialized: now 1786223171620934 us; error 35 us; skew 500 ppm
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[7902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223171, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:11.623379 7915 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:11.623859 7915 webserver.cc:460] Webserver started at http://127.4.177.126:42503/ using document root <none> and password file <none>
I20260808 21:06:11.624110 7915 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:11.624182 7915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:11.624334 7915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:11.625391 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "d31a692b786b4ac8b94f82f941bde061"
format_stamp: "Formatted at 2026-08-08 21:06:11 on dist-test-slave-chsk"
server_key: "bc6b3a60aceddf7d5c65dda05b36064a"
server_key_iv: "251686445c681b626191e96d2e2e1644"
server_key_version: "encryptionkey@0"
I20260808 21:06:11.625761 7915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "d31a692b786b4ac8b94f82f941bde061"
format_stamp: "Formatted at 2026-08-08 21:06:11 on dist-test-slave-chsk"
server_key: "bc6b3a60aceddf7d5c65dda05b36064a"
server_key_iv: "251686445c681b626191e96d2e2e1644"
server_key_version: "encryptionkey@0"
I20260808 21:06:11.627074 7915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:06:11.627826 7932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:11.628001 7915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:11.628095 7915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "d31a692b786b4ac8b94f82f941bde061"
format_stamp: "Formatted at 2026-08-08 21:06:11 on dist-test-slave-chsk"
server_key: "bc6b3a60aceddf7d5c65dda05b36064a"
server_key_iv: "251686445c681b626191e96d2e2e1644"
server_key_version: "encryptionkey@0"
I20260808 21:06:11.628180 7915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:11.637570 7915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:11.639333 7915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:11.639489 7915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:11.643740 7915 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:38919
I20260808 21:06:11.643774 7990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:38919 every 8 connection(s)
I20260808 21:06:11.644112 7915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:11.644937 7992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:11.647275 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061: Bootstrap starting.
I20260808 21:06:11.647881 7992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:11.648190 7992 log.cc:824] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:11.648926 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061: No bootstrap required, opened a new log
I20260808 21:06:11.650804 7992 raft_consensus.cc:348] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } }
I20260808 21:06:11.650962 7992 raft_consensus.cc:374] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:11.651012 7992 raft_consensus.cc:729] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31a692b786b4ac8b94f82f941bde061, State: Initialized, Role: FOLLOWER
I20260808 21:06:11.651206 7992 consensus_queue.cc:260] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } }
I20260808 21:06:11.651304 7992 raft_consensus.cc:388] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:11.651358 7992 raft_consensus.cc:482] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:11.651409 7992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:11.652230 7992 raft_consensus.cc:504] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } }
I20260808 21:06:11.652357 7992 leader_election.cc:302] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d31a692b786b4ac8b94f82f941bde061; no voters:
I20260808 21:06:11.652544 7992 leader_election.cc:288] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:11.652554 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 7915
I20260808 21:06:11.652624 7996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:11.652655 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:11.652789 7996 raft_consensus.cc:686] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [term 1 LEADER]: Becoming Leader. State: Replica: d31a692b786b4ac8b94f82f941bde061, State: Running, Role: LEADER
I20260808 21:06:11.652866 4805 external_mini_cluster.cc:1428] Setting key 9641104a86c7f557764ff78a711c2c60
I20260808 21:06:11.652942 7996 consensus_queue.cc:237] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } }
I20260808 21:06:11.652827 7992 sys_catalog.cc:564] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:11.653475 7999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d31a692b786b4ac8b94f82f941bde061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } } }
I20260808 21:06:11.653786 7999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:11.653674 7998 sys_catalog.cc:455] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d31a692b786b4ac8b94f82f941bde061. Latest consensus state: current_term: 1 leader_uuid: "d31a692b786b4ac8b94f82f941bde061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d31a692b786b4ac8b94f82f941bde061" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38919 } } }
I20260808 21:06:11.654070 8010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:11.654711 8010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:11.654080 7998 sys_catalog.cc:458] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:11.656507 8010 catalog_manager.cc:1347] Generated new cluster ID: 3bf1a748875341edb1803c50eb7683de
I20260808 21:06:11.656559 8010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[7902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:11.665221 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:11.653289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.4.177.126:38919)
0808 21:06:11.653463 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:11.653473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:11.654128 (+ 655us) server_negotiation.cc:407] Connection header received
0808 21:06:11.654333 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:11.654341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:11.654431 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:11.654576 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:11.655411 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:11.656835 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:11.657846 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:11.658442 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:11.658526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:11.661335 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:11.661339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:11.661339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:11.661352 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:11.663340 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:11.663713 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:11.663716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:11.663716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:11.663812 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:11.663983 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:11.663984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:11.663984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:11.664262 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:11.664416 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:11.664831 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 21:06:11.664966 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260808 21:06:11.666163 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:06:11.681017 8010 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:11.681856 8010 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:11.686311 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:11.682033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.4.177.126:38919)
0808 21:06:11.682075 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:11.682077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:11.682152 (+ 75us) server_negotiation.cc:407] Connection header received
0808 21:06:11.682247 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:11.682247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:11.682257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:11.682320 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:11.682762 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:11.683562 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:11.684656 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:11.684986 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:11.685045 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:11.685827 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:11.685828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:11.685828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:11.686059 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:11.686078 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:11.686085 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:11.686144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:11.686211 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 21:06:11.686239 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:06:11.687001 8010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d31a692b786b4ac8b94f82f941bde061: Generated new TSK 0
I20260808 21:06:11.687216 8010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:11.692660 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 7915
2026-08-08T21:06:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1358 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[8031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[8031](info): set up 2 sockets
Aug 08 21:06:11 dist-test-slave-chsk krb5kdc[8031](info): commencing operation
krb5kdc: starting...
W20260808 21:06:12.778762 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:12 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:12.901575 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43055
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:12.985319 8044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:12.985548 8044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:12.985572 8044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:12.986766 8044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:12.986811 8044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:12.986826 8044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:12.986840 8044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:12.986889 8044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:12.988463 8044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:43055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:12.988726 8044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:12.988969 8044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:12.994395 8050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:12.994412 8049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:12.994385 8052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:12.994477 8044 server_base.cc:1034] running on GCE node
I20260808 21:06:12.994863 8044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:12.995261 8044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:12.996420 8044 hybrid_clock.cc:648] HybridClock initialized: now 1786223172996403 us; error 43 us; skew 500 ppm
Aug 08 21:06:12 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223172, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:12.999059 8044 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:12.999588 8044 webserver.cc:460] Webserver started at http://127.4.177.126:46039/ using document root <none> and password file <none>
I20260808 21:06:12.999783 8044 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:12.999825 8044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:12.999918 8044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:13.000970 8044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "ccae0e61f8a14b6abd98b655bd8e45c7"
format_stamp: "Formatted at 2026-08-08 21:06:12 on dist-test-slave-chsk"
server_key: "0eb7445b09220b6c46731e83e2b80da9"
server_key_iv: "9833576e496fadcda9ca909a9462d4f0"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.001289 8044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "ccae0e61f8a14b6abd98b655bd8e45c7"
format_stamp: "Formatted at 2026-08-08 21:06:12 on dist-test-slave-chsk"
server_key: "0eb7445b09220b6c46731e83e2b80da9"
server_key_iv: "9833576e496fadcda9ca909a9462d4f0"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.002717 8044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:13.003551 8059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.003756 8044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:13.003854 8044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "ccae0e61f8a14b6abd98b655bd8e45c7"
format_stamp: "Formatted at 2026-08-08 21:06:12 on dist-test-slave-chsk"
server_key: "0eb7445b09220b6c46731e83e2b80da9"
server_key_iv: "9833576e496fadcda9ca909a9462d4f0"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.003913 8044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:13.023314 8044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:13.025185 8044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:13.025300 8044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:13.029234 8044 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:43055
I20260808 21:06:13.029345 8117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:43055 every 8 connection(s)
I20260808 21:06:13.029667 8044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:13.030834 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:13.033124 8118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7: Bootstrap starting.
I20260808 21:06:13.033747 8118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:13.034047 8118 log.cc:824] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:13.034771 8118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7: No bootstrap required, opened a new log
I20260808 21:06:13.036470 8118 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } }
I20260808 21:06:13.036626 8118 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:13.036648 8118 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccae0e61f8a14b6abd98b655bd8e45c7, State: Initialized, Role: FOLLOWER
I20260808 21:06:13.036854 8118 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } }
I20260808 21:06:13.036938 8118 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:13.037012 8118 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:13.037062 8118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:13.037214 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8044
I20260808 21:06:13.037325 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:13.037515 4805 external_mini_cluster.cc:1428] Setting key 249d6e71230821466c5934a9c8922783
I20260808 21:06:13.038166 8118 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } }
I20260808 21:06:13.038277 8118 leader_election.cc:302] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccae0e61f8a14b6abd98b655bd8e45c7; no voters:
I20260808 21:06:13.038441 8118 leader_election.cc:288] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:13.038491 8123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:13.038650 8123 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [term 1 LEADER]: Becoming Leader. State: Replica: ccae0e61f8a14b6abd98b655bd8e45c7, State: Running, Role: LEADER
I20260808 21:06:13.038691 8118 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:13.038775 8123 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } }
I20260808 21:06:13.039224 8123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccae0e61f8a14b6abd98b655bd8e45c7. Latest consensus state: current_term: 1 leader_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } } }
I20260808 21:06:13.039358 8123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:13.039389 8124 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccae0e61f8a14b6abd98b655bd8e45c7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43055 } } }
I20260808 21:06:13.039764 8124 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:13.040038 8129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:13.040771 8129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:13.042330 8129 catalog_manager.cc:1347] Generated new cluster ID: 3d5827f476934befa6faa24c3f0ee083
I20260808 21:06:13.042385 8129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:13.050423 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.038048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.4.177.126:43055)
0808 21:06:13.038249 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:13.038259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:13.038298 (+ 39us) server_negotiation.cc:407] Connection header received
0808 21:06:13.038419 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:13.038428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:13.038488 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:13.038634 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:13.039664 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.041330 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.042378 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.043022 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.043133 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.045932 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:13.045936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:13.045937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:13.045949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:13.048756 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.049152 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.049155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.049155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.049280 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.049487 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.049488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.049488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.049816 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:13.049989 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:13.050057 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 21:06:13.050196 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260808 21:06:13.057577 8129 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:13.058307 8129 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:13.069957 8129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccae0e61f8a14b6abd98b655bd8e45c7: Generated new TSK 0
I20260808 21:06:13.070175 8129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:13.106107 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:13.188004 8148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:13.188195 8148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:13.188218 8148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:13.189334 8148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:13.189378 8148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:13.189432 8148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:13.190862 8148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:13.191092 8148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:13.191339 8148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:06:13.196132 8148 server_base.cc:1034] running on GCE node
W20260808 21:06:13.196117 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.196116 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.196108 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:13.196506 8148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:13.196770 8148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:13.197952 8148 hybrid_clock.cc:648] HybridClock initialized: now 1786223173197927 us; error 36 us; skew 500 ppm
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:13.200553 8148 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:13.200968 8148 webserver.cc:460] Webserver started at http://127.4.177.65:46361/ using document root <none> and password file <none>
I20260808 21:06:13.201190 8148 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:13.201267 8148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:13.201421 8148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:13.202427 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "7ea4b736a53c4d2a9358331e58272a87"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "5a8575d099ce2754c345a54163e5945a"
server_key_iv: "91cf8af31ba72f1cbfa95feae5aa255e"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.202800 8148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "7ea4b736a53c4d2a9358331e58272a87"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "5a8575d099ce2754c345a54163e5945a"
server_key_iv: "91cf8af31ba72f1cbfa95feae5aa255e"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.204177 8148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:13.204905 8165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.205076 8148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.205188 8148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "7ea4b736a53c4d2a9358331e58272a87"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "5a8575d099ce2754c345a54163e5945a"
server_key_iv: "91cf8af31ba72f1cbfa95feae5aa255e"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.205269 8148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:13.227026 8148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:13.229532 8148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:13.229712 8148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:13.229971 8148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:13.230309 8148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:13.230365 8148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.230400 8148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:13.230445 8148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.236341 8148 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:39727
I20260808 21:06:13.236394 8291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:39727 every 8 connection(s)
I20260808 21:06:13.236716 8148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:13.241791 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8148
I20260808 21:06:13.241904 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:13.242138 4805 external_mini_cluster.cc:1428] Setting key 70af5ffab3e40d7ee96f8f6b49cfbe70
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:13.246537 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.237436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:40917 (local address 127.4.177.126:43055)
0808 21:06:13.237514 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:13.237516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:13.237670 (+ 154us) server_negotiation.cc:407] Connection header received
0808 21:06:13.238184 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:13.238185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:13.238195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:13.238261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:13.239368 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.239879 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.240974 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.241233 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.241289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.243517 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:13.243518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:13.243519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:13.243526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:13.245061 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.245590 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.245591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.245591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.245642 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.245926 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.245927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.245927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.246025 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:13.246076 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:13.246408 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 21:06:13.246471 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 21:06:13.246850 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.237381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43055 (local address 127.4.177.65:40917)
0808 21:06:13.237546 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:13.237588 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:13.238111 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:13.238331 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:13.238334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:13.239002 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:13.239281 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.239287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.239963 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.239963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.240904 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.240906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.241306 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.241307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.241406 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.241894 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:13.241905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:13.243426 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:13.245146 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.245154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.245157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.245512 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.245709 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.245710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.245710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.245860 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.246089 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:13.246096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:13.246274 (+ 178us) client_negotiation.cc:769] Sending connection context
0808 21:06:13.246389 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 21:06:13.246564 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260808 21:06:13.247473 8292 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43055
I20260808 21:06:13.247586 8292 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:13.247795 8292 heartbeater.cc:511] Master 127.4.177.126:43055 requested a full tablet report, sending...
I20260808 21:06:13.248337 8078 ts_manager.cc:194] Registered new tserver with Master: 7ea4b736a53c4d2a9358331e58272a87 (127.4.177.65:39727)
I20260808 21:06:13.248946 8078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:40917
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:13.269640 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:13.351383 8298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:13.351610 8298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:13.351649 8298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:13.352762 8298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:13.352828 8298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:13.352872 8298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:13.354323 8298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:13.354580 8298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:13.354818 8298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:13.359656 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.359664 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.359678 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:13.360038 8298 server_base.cc:1034] running on GCE node
I20260808 21:06:13.360255 8298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:13.360517 8298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:13.361680 8298 hybrid_clock.cc:648] HybridClock initialized: now 1786223173361666 us; error 24 us; skew 500 ppm
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:13.364125 8298 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:13.364607 8298 webserver.cc:460] Webserver started at http://127.4.177.66:37051/ using document root <none> and password file <none>
I20260808 21:06:13.364856 8298 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:13.364925 8298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:13.365088 8298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:13.366122 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "916a0aee442c455bade2c399f6e11394"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "621a84d9278826697f97957881e3ce24"
server_key_iv: "865806a17e1ce10ee0b6f52c5e054fdb"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.366499 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "916a0aee442c455bade2c399f6e11394"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "621a84d9278826697f97957881e3ce24"
server_key_iv: "865806a17e1ce10ee0b6f52c5e054fdb"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.367923 8298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:13.368654 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.368819 8298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:13.368920 8298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "916a0aee442c455bade2c399f6e11394"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "621a84d9278826697f97957881e3ce24"
server_key_iv: "865806a17e1ce10ee0b6f52c5e054fdb"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.369001 8298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:13.385494 8298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:13.387456 8298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:13.387638 8298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:13.387902 8298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:13.388233 8298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:13.388290 8298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.388334 8298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:13.388379 8298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.394281 8298 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:45521
I20260808 21:06:13.394351 8432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:45521 every 8 connection(s)
I20260808 21:06:13.394703 8298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:13.395323 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8298
I20260808 21:06:13.395464 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:13.395684 4805 external_mini_cluster.cc:1428] Setting key 4830aef30da20c4355bdbf52abc9e40e
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
I20260808 21:06:13.405658 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.395473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:39303 (local address 127.4.177.126:43055)
0808 21:06:13.395529 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:13.395531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:13.395797 (+ 266us) server_negotiation.cc:407] Connection header received
0808 21:06:13.396434 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:13.396435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:13.396448 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:13.396512 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:13.397864 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.398410 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.399970 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.400156 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.400210 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.402727 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:13.402728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:13.402729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:13.402737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:13.404204 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.404641 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.404642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.404643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.404706 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.404965 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.404966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.404966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.405089 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:13.405167 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:13.405447 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 21:06:13.405538 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 21:06:13.405831 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.395428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43055 (local address 127.4.177.66:39303)
0808 21:06:13.395614 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:13.395666 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:13.396331 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:13.396592 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:13.396597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:13.397480 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:13.397770 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.397776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.398517 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.398518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.399886 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.399889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.400236 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.400237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.400390 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.400930 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:13.400941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:13.402619 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:13.404317 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.404325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.404327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.404561 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.404788 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.404789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.404789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.404899 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.405156 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:13.405163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:13.405309 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 21:06:13.405413 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 21:06:13.405584 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260808 21:06:13.406512 8435 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43055
I20260808 21:06:13.406643 8435 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:13.406847 8435 heartbeater.cc:511] Master 127.4.177.126:43055 requested a full tablet report, sending...
I20260808 21:06:13.407483 8078 ts_manager.cc:194] Registered new tserver with Master: 916a0aee442c455bade2c399f6e11394 (127.4.177.66:45521)
I20260808 21:06:13.407847 8078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:39303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:13.421833 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:13.504194 8441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:13.504407 8441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:13.504431 8441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:13.505558 8441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:13.505600 8441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:13.505649 8441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:13.507474 8441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:13.507717 8441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:13.507961 8441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:13.513043 8447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.513083 8449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:13.513051 8446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:13.513360 8441 server_base.cc:1034] running on GCE node
I20260808 21:06:13.513559 8441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:13.513860 8441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:13.515029 8441 hybrid_clock.cc:648] HybridClock initialized: now 1786223173515009 us; error 35 us; skew 500 ppm
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:13.522604 8441 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:13.523119 8441 webserver.cc:460] Webserver started at http://127.4.177.67:43321/ using document root <none> and password file <none>
I20260808 21:06:13.523412 8441 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:13.523483 8441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:13.523625 8441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:13.524706 8441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "578b08dff1994d92bef3d623a1903ab7"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "a6a82dc581f6a0021c40ed6fe5b8715b"
server_key_iv: "3a0e1d3d107b02aad46d0a2b104a99e2"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.525084 8441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "578b08dff1994d92bef3d623a1903ab7"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "a6a82dc581f6a0021c40ed6fe5b8715b"
server_key_iv: "3a0e1d3d107b02aad46d0a2b104a99e2"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.526508 8441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:13.527292 8458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.527478 8441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:13.527578 8441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "578b08dff1994d92bef3d623a1903ab7"
format_stamp: "Formatted at 2026-08-08 21:06:13 on dist-test-slave-chsk"
server_key: "a6a82dc581f6a0021c40ed6fe5b8715b"
server_key_iv: "3a0e1d3d107b02aad46d0a2b104a99e2"
server_key_version: "encryptionkey@0"
I20260808 21:06:13.527659 8441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:13.541738 8441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:13.543685 8441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:13.543850 8441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:13.544078 8441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:13.544409 8441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:13.544464 8441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.544499 8441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:13.544526 8441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:13.550443 8441 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40631
I20260808 21:06:13.550472 8584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40631 every 8 connection(s)
I20260808 21:06:13.550844 8441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:13.557318 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8441
I20260808 21:06:13.557405 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:13.557615 4805 external_mini_cluster.cc:1428] Setting key 8c8207efabdc8a28366ac745cf925b71
I20260808 21:06:13.560473 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.551536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:60947 (local address 127.4.177.126:43055)
0808 21:06:13.551600 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:13.551602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:13.551777 (+ 175us) server_negotiation.cc:407] Connection header received
0808 21:06:13.552341 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:13.552343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:13.552355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:13.552421 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:13.553327 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.553830 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.554757 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.554932 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.554982 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.557061 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:13.557062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:13.557062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:13.557069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:13.558767 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.559326 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.559327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.559328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.559389 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.559658 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.559659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.559660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.559794 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:13.559860 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:13.560260 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 21:06:13.560362 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:06:13.560570 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.551489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43055 (local address 127.4.177.67:60947)
0808 21:06:13.551650 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:13.551691 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:13.552156 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:13.552488 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:13.552492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:13.553052 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:13.553246 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.553251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.553906 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.553907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.554686 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:13.554688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.555002 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.555003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:13.555096 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.555606 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:13.555613 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:13.556975 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:13.558839 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.558846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.558848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.559249 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.559464 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:13.559465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:13.559465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:13.559599 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:13.559894 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:13.559900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:13.560065 (+ 165us) client_negotiation.cc:769] Sending connection context
0808 21:06:13.560188 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 21:06:13.560355 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260808 21:06:13.561028 8585 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43055
I20260808 21:06:13.561108 8585 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:13.561281 8585 heartbeater.cc:511] Master 127.4.177.126:43055 requested a full tablet report, sending...
I20260808 21:06:13.561643 8078 ts_manager.cc:194] Registered new tserver with Master: 578b08dff1994d92bef3d623a1903ab7 (127.4.177.67:40631)
I20260808 21:06:13.561972 8078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:60947
I20260808 21:06:13.569593 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223173, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:13.592617 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:13.585910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.4.177.126:43055)
0808 21:06:13.585979 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:13.585982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:13.586101 (+ 119us) server_negotiation.cc:407] Connection header received
0808 21:06:13.586148 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:13.586149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:13.586158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:13.586213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:13.586747 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.587269 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.587960 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:13.588142 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:13.588189 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:13.590294 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:13.590296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:13.590296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:13.590302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:13.591687 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.591974 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.591974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.591975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.592023 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:13.592165 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:13.592165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:13.592166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:13.592294 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:13.592361 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:13.592483 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 21:06:13.592551 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260808 21:06:13.592839 8078 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55798
I20260808 21:06:13.593375 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8148
I20260808 21:06:13.601681 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8298
I20260808 21:06:13.607553 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8441
I20260808 21:06:13.613507 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8044
2026-08-08T21:06:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1924 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8598](info): set up 2 sockets
Aug 08 21:06:13 dist-test-slave-chsk krb5kdc[8598](info): commencing operation
krb5kdc: starting...
W20260808 21:06:14.695240 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:14 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:14.815913 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:45785
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:14.897076 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:14.897279 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:14.897300 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:14.898375 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:14.898406 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:14.898420 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:14.898433 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:14.898445 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:14.899870 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:14.900063 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:14.900229 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:14.905054 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:14.905072 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:14.905074 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:14.905344 8613 server_base.cc:1034] running on GCE node
I20260808 21:06:14.905606 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:14.905918 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:14.907081 8613 hybrid_clock.cc:648] HybridClock initialized: now 1786223174907076 us; error 37 us; skew 500 ppm
Aug 08 21:06:14 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223174, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:14.909507 8613 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:14.909978 8613 webserver.cc:460] Webserver started at http://127.4.177.126:45167/ using document root <none> and password file <none>
I20260808 21:06:14.910205 8613 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:14.910274 8613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:14.910431 8613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:14.911620 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "a82133554dc64a4d89577aad7af0aa8f"
format_stamp: "Formatted at 2026-08-08 21:06:14 on dist-test-slave-chsk"
server_key: "70d46c588674a78378391ad4bae715e9"
server_key_iv: "d0a0968a48ee601e5ec8defe53466d5e"
server_key_version: "encryptionkey@0"
I20260808 21:06:14.911984 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "a82133554dc64a4d89577aad7af0aa8f"
format_stamp: "Formatted at 2026-08-08 21:06:14 on dist-test-slave-chsk"
server_key: "70d46c588674a78378391ad4bae715e9"
server_key_iv: "d0a0968a48ee601e5ec8defe53466d5e"
server_key_version: "encryptionkey@0"
I20260808 21:06:14.913364 8613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:14.914099 8630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:14.914280 8613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:14.914381 8613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "a82133554dc64a4d89577aad7af0aa8f"
format_stamp: "Formatted at 2026-08-08 21:06:14 on dist-test-slave-chsk"
server_key: "70d46c588674a78378391ad4bae715e9"
server_key_iv: "d0a0968a48ee601e5ec8defe53466d5e"
server_key_version: "encryptionkey@0"
I20260808 21:06:14.914465 8613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:14.924525 8613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:14.926205 8613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:14.926359 8613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:14.930431 8613 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:45785
I20260808 21:06:14.930486 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:45785 every 8 connection(s)
I20260808 21:06:14.930992 8613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:14.931751 8688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:14.933928 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f: Bootstrap starting.
I20260808 21:06:14.934453 8688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:14.934708 8688 log.cc:824] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:14.935429 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f: No bootstrap required, opened a new log
I20260808 21:06:14.937064 8688 raft_consensus.cc:348] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } }
I20260808 21:06:14.937203 8688 raft_consensus.cc:374] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:14.937227 8688 raft_consensus.cc:729] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82133554dc64a4d89577aad7af0aa8f, State: Initialized, Role: FOLLOWER
I20260808 21:06:14.937355 8688 consensus_queue.cc:260] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } }
I20260808 21:06:14.937436 8688 raft_consensus.cc:388] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:14.937458 8688 raft_consensus.cc:482] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:14.937486 8688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:14.938344 8688 raft_consensus.cc:504] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } }
I20260808 21:06:14.938441 8688 leader_election.cc:302] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a82133554dc64a4d89577aad7af0aa8f; no voters:
I20260808 21:06:14.938587 8688 leader_election.cc:288] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:14.938627 8691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:14.938764 8691 raft_consensus.cc:686] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [term 1 LEADER]: Becoming Leader. State: Replica: a82133554dc64a4d89577aad7af0aa8f, State: Running, Role: LEADER
I20260808 21:06:14.938843 8688 sys_catalog.cc:564] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:14.938840 8691 consensus_queue.cc:237] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } }
I20260808 21:06:14.939273 8692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a82133554dc64a4d89577aad7af0aa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } } }
I20260808 21:06:14.939311 8693 sys_catalog.cc:455] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a82133554dc64a4d89577aad7af0aa8f. Latest consensus state: current_term: 1 leader_uuid: "a82133554dc64a4d89577aad7af0aa8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a82133554dc64a4d89577aad7af0aa8f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 45785 } } }
I20260808 21:06:14.939440 8692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:14.939459 8693 sys_catalog.cc:458] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:14.939816 8701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:14.940383 8701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:14.941139 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8613
I20260808 21:06:14.941243 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:14.941435 4805 external_mini_cluster.cc:1428] Setting key 5afe4672ac5e8da9521330fe90cd3fc3
I20260808 21:06:14.942044 8701 catalog_manager.cc:1347] Generated new cluster ID: fe09b3c47ecd4d1682b4169c8aacc7ae
I20260808 21:06:14.942112 8701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:14 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:14.954272 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:14.941918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.4.177.126:45785)
0808 21:06:14.943264 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:14.943273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:14.943465 (+ 192us) server_negotiation.cc:407] Connection header received
0808 21:06:14.943616 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:14.943624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:14.943691 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:14.943849 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:14.944541 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:14.945997 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:14.947010 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:14.947760 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:14.947875 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:14.950666 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:14.950670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:14.950670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:14.950682 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:14.952625 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:14.953035 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:14.953038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:14.953038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:14.953143 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:14.953367 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:14.953367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:14.953368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:14.953645 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:14.953775 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:14.953921 (+ 146us) server_negotiation.cc:299] Negotiation successful
0808 21:06:14.954043 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":71,"threads_started":1}
I20260808 21:06:14.959125 8701 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:14.959753 8701 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:14.963819 8701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a82133554dc64a4d89577aad7af0aa8f: Generated new TSK 0
I20260808 21:06:14.963975 8701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:14.996227 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:15.079716 8716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:15.079965 8716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:15.080016 8716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:15.081249 8716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:15.081310 8716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:15.081357 8716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:15.082882 8716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:15.083217 8716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:15.083453 8716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:15.088341 8721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:15.088341 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:15.088598 8716 server_base.cc:1034] running on GCE node
W20260808 21:06:15.088395 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:15.088873 8716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:15.089123 8716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:15.090323 8716 hybrid_clock.cc:648] HybridClock initialized: now 1786223175090311 us; error 26 us; skew 500 ppm
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:15.092921 8716 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:15.093551 8716 webserver.cc:460] Webserver started at http://127.4.177.65:40417/ using document root <none> and password file <none>
I20260808 21:06:15.093781 8716 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:15.093852 8716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:15.094005 8716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:15.095016 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "a18f2524feec4a229cbb3ae536093930"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "5edd4c0d99c116775f24573407eede4a"
server_key_iv: "9b0175cd855f9ae2833ed91f1913e95e"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.095474 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "a18f2524feec4a229cbb3ae536093930"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "5edd4c0d99c116775f24573407eede4a"
server_key_iv: "9b0175cd855f9ae2833ed91f1913e95e"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.096797 8716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:15.097499 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.097666 8716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:15.097774 8716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "a18f2524feec4a229cbb3ae536093930"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "5edd4c0d99c116775f24573407eede4a"
server_key_iv: "9b0175cd855f9ae2833ed91f1913e95e"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.097855 8716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:15.127971 8716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:15.130089 8716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:15.130272 8716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:15.130542 8716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:15.130884 8716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:15.130939 8716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.130983 8716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:15.131007 8716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.137140 8716 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35129
I20260808 21:06:15.137190 8852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35129 every 8 connection(s)
I20260808 21:06:15.137504 8716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:15.141976 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8716
I20260808 21:06:15.142109 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:15.142305 4805 external_mini_cluster.cc:1428] Setting key 74f76627b3eb3c5d750e7d1e2dc4f460
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:15.147806 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.138262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:50507 (local address 127.4.177.126:45785)
0808 21:06:15.138338 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:15.138347 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:15.138550 (+ 203us) server_negotiation.cc:407] Connection header received
0808 21:06:15.139548 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:15.139549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:15.139585 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:15.139671 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:15.140323 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.140908 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.141901 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.142233 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.142309 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.144583 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:15.144585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:15.144585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:15.144594 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:15.146287 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.146698 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.146699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.146700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.146767 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.147003 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.147004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.147005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.147191 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:15.147284 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:15.147574 (+ 290us) server_negotiation.cc:299] Negotiation successful
0808 21:06:15.147676 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:15.147830 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.138224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45785 (local address 127.4.177.65:50507)
0808 21:06:15.138443 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:15.138483 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:15.139459 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:15.139751 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:15.139754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:15.140007 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:15.140245 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.140250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.140996 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.140997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.141807 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.141809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.142331 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.142332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.142489 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.143047 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:15.143058 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:15.144484 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:15.146381 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.146387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.146389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.146621 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.146837 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.146837 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.146837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.146935 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.147272 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:15.147277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:15.147389 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 21:06:15.147530 (+ 141us) client_negotiation.cc:240] Negotiation successful
0808 21:06:15.147664 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":117,"threads_started":1}
I20260808 21:06:15.148329 8853 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45785
I20260808 21:06:15.148440 8853 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:15.148653 8853 heartbeater.cc:511] Master 127.4.177.126:45785 requested a full tablet report, sending...
I20260808 21:06:15.149189 8649 ts_manager.cc:194] Registered new tserver with Master: a18f2524feec4a229cbb3ae536093930 (127.4.177.65:35129)
I20260808 21:06:15.149817 8649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:50507
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:15.169875 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:15.250265 8858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:15.250453 8858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:15.250475 8858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:15.251646 8858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:15.251688 8858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:15.251740 8858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:15.253260 8858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:15.253496 8858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:15.253728 8858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:15.258524 8864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:15.258530 8866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:15.258530 8863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:15.258808 8858 server_base.cc:1034] running on GCE node
I20260808 21:06:15.258982 8858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:15.259219 8858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:15.260362 8858 hybrid_clock.cc:648] HybridClock initialized: now 1786223175260352 us; error 26 us; skew 500 ppm
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:15.262789 8858 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:15.263305 8858 webserver.cc:460] Webserver started at http://127.4.177.66:37363/ using document root <none> and password file <none>
I20260808 21:06:15.263499 8858 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:15.263540 8858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:15.263624 8858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:15.264557 8858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "7709476162c64d48bceed8b70d6fb3f5"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "940a4a44a2396a2da0c849db81491f3c"
server_key_iv: "3f843d8c416753208cd36d035a3a5ae8"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.264842 8858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "7709476162c64d48bceed8b70d6fb3f5"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "940a4a44a2396a2da0c849db81491f3c"
server_key_iv: "3f843d8c416753208cd36d035a3a5ae8"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.266069 8858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:15.266727 8874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.266881 8858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:15.266968 8858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "7709476162c64d48bceed8b70d6fb3f5"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "940a4a44a2396a2da0c849db81491f3c"
server_key_iv: "3f843d8c416753208cd36d035a3a5ae8"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.267019 8858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:15.274631 8858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:15.276348 8858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:15.276451 8858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:15.276607 8858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:15.276837 8858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:15.276870 8858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.276889 8858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:15.276921 8858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.282503 8858 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:36319
I20260808 21:06:15.282595 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:36319 every 8 connection(s)
I20260808 21:06:15.282869 8858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:15.284341 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 8858
I20260808 21:06:15.284461 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:15.284647 4805 external_mini_cluster.cc:1428] Setting key be20606e881340078ae263f1ab633516
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:15.293368 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.283631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:34055 (local address 127.4.177.126:45785)
0808 21:06:15.283700 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:15.283702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:15.283910 (+ 208us) server_negotiation.cc:407] Connection header received
0808 21:06:15.284987 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:15.284988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:15.285000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:15.285067 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:15.285730 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.286485 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.287804 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.287990 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.288048 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.290463 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:15.290464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:15.290465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:15.290474 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:15.291944 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.292382 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.292384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.292384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.292446 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.292700 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.292701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.292701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.292822 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:15.292903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:15.293173 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 21:06:15.293276 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 21:06:15.293550 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.283583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45785 (local address 127.4.177.66:34055)
0808 21:06:15.283789 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:15.283828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:15.284847 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:15.285156 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:15.285160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:15.285464 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:15.285640 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.285644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.286577 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.286578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.287712 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.287715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.288075 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.288076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.288235 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.288733 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:15.288744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:15.290364 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:15.292045 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.292052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.292055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.292279 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.292515 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.292516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.292517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.292628 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.292893 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:15.292900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:15.293040 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 21:06:15.293152 (+ 112us) client_negotiation.cc:240] Negotiation successful
0808 21:06:15.293309 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
I20260808 21:06:15.294175 9007 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45785
I20260808 21:06:15.294283 9007 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:15.294458 9007 heartbeater.cc:511] Master 127.4.177.126:45785 requested a full tablet report, sending...
I20260808 21:06:15.294907 8649 ts_manager.cc:194] Registered new tserver with Master: 7709476162c64d48bceed8b70d6fb3f5 (127.4.177.66:36319)
I20260808 21:06:15.295295 8649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:34055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:15.311410 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:15.392077 9012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:15.392283 9012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:15.392306 9012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:15.393369 9012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:15.393402 9012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:15.393428 9012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:15.394846 9012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:45785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:15.395037 9012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:15.395296 9012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:15.400064 9018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:15.400064 9017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:15.400143 9020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:15.400472 9012 server_base.cc:1034] running on GCE node
I20260808 21:06:15.400704 9012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:15.400972 9012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:15.402112 9012 hybrid_clock.cc:648] HybridClock initialized: now 1786223175402092 us; error 37 us; skew 500 ppm
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:15.404738 9012 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:15.405121 9012 webserver.cc:460] Webserver started at http://127.4.177.67:34555/ using document root <none> and password file <none>
I20260808 21:06:15.405305 9012 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:15.405346 9012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:15.405435 9012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:15.406569 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "eed502c54b004750a8f0e9baa0ab08b7"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "d7e58151bda74c54b8be90690b31f4c7"
server_key_iv: "8a08ca1f3a491c29d1bfb9e606eecc65"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.406955 9012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "eed502c54b004750a8f0e9baa0ab08b7"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "d7e58151bda74c54b8be90690b31f4c7"
server_key_iv: "8a08ca1f3a491c29d1bfb9e606eecc65"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.408353 9012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:15.409133 9028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.409332 9012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:15.409435 9012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "eed502c54b004750a8f0e9baa0ab08b7"
format_stamp: "Formatted at 2026-08-08 21:06:15 on dist-test-slave-chsk"
server_key: "d7e58151bda74c54b8be90690b31f4c7"
server_key_iv: "8a08ca1f3a491c29d1bfb9e606eecc65"
server_key_version: "encryptionkey@0"
I20260808 21:06:15.409515 9012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:15.421975 9012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:15.424084 9012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:15.424252 9012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:15.424486 9012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:15.424814 9012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:15.424870 9012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.424913 9012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:15.424962 9012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:15.432018 9012 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:45181
I20260808 21:06:15.432070 9153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:45181 every 8 connection(s)
I20260808 21:06:15.432403 9012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:15.436899 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9012
I20260808 21:06:15.437011 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:15.437203 4805 external_mini_cluster.cc:1428] Setting key fdcfab7b978d667e9294ba43211bdeed
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:15.442648 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.433139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:40703 (local address 127.4.177.126:45785)
0808 21:06:15.433190 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:15.433192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:15.433392 (+ 200us) server_negotiation.cc:407] Connection header received
0808 21:06:15.434406 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:15.434407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:15.434421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:15.434488 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:15.435279 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.435793 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.436909 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.437189 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.437271 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.439971 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:15.439973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:15.439973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:15.439980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:15.441428 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.441814 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.441815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.441815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.441876 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.442107 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.442108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.442108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.442224 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:15.442278 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:15.442514 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 21:06:15.442581 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:06:15.442809 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.433117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:45785 (local address 127.4.177.67:40703)
0808 21:06:15.433266 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:15.433304 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:15.434288 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:15.434565 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:15.434568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:15.434892 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:15.435091 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.435096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.435923 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.435924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.436789 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:15.436792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.437280 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.437281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:15.437422 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.438042 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:15.438050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:15.439876 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:15.441503 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.441509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.441511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.441724 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.441942 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:15.441943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:15.441943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:15.442049 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:15.442283 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:15.442287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:15.442401 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 21:06:15.442483 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 21:06:15.442604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260808 21:06:15.443310 9154 heartbeater.cc:347] Connected to a master server at 127.4.177.126:45785
I20260808 21:06:15.443423 9154 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:15.443604 9154 heartbeater.cc:511] Master 127.4.177.126:45785 requested a full tablet report, sending...
I20260808 21:06:15.443986 8649 ts_manager.cc:194] Registered new tserver with Master: eed502c54b004750a8f0e9baa0ab08b7 (127.4.177.67:45181)
I20260808 21:06:15.444360 8649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:40703
I20260808 21:06:15.449227 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:06:15.473357 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.466010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.4.177.65:35129)
0808 21:06:15.466192 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:15.466196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:15.466206 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:15.466232 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:15.466234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:15.466248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:15.466326 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:15.466874 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.467649 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.468349 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.468642 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.468695 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.470921 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:15.470929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:15.470929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:15.470935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:15.472326 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.472659 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.472661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.472662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.472714 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.472872 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.472872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.472873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.472994 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:15.473101 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:15.473160 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 21:06:15.473218 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[8598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:15.482112 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:15.474690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.4.177.126:45785)
0808 21:06:15.474745 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:15.474748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:15.474757 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:15.474830 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:15.474831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:15.474841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:15.474925 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:15.475484 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.475978 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.476626 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:15.476838 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:15.476914 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:15.479680 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:15.479681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:15.479682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:15.479690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:15.480908 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.481276 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.481277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.481278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.481336 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:15.481529 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:15.481530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:15.481530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:15.481697 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:15.481800 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:15.481910 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 21:06:15.481990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260808 21:06:15.482362 8649 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40864
I20260808 21:06:15.482733 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8716
I20260808 21:06:15.490823 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8858
I20260808 21:06:15.496603 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9012
I20260808 21:06:15.501641 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 8613
2026-08-08T21:06:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1887 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[9166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[9166](info): set up 2 sockets
Aug 08 21:06:15 dist-test-slave-chsk krb5kdc[9166](info): commencing operation
krb5kdc: starting...
W20260808 21:06:16.579967 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:16 dist-test-slave-chsk krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:16.699244 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40825
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:16.781569 9181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:16.781765 9181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:16.781788 9181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:16.782878 9181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:16.782913 9181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:16.782928 9181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:16.782940 9181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:16.782953 9181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:16.784394 9181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:40825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:16.784571 9181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:16.784732 9181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:16.789359 9187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:16.789554 9181 server_base.cc:1034] running on GCE node
W20260808 21:06:16.789347 9186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:16.789347 9189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:16.789911 9181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:16.790242 9181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:16.791384 9181 hybrid_clock.cc:648] HybridClock initialized: now 1786223176791368 us; error 40 us; skew 500 ppm
Aug 08 21:06:16 dist-test-slave-chsk krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223176, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:16.793856 9181 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:16.794188 9181 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:16.794234 9181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:16.794378 9181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:16.795478 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "f4b02bacdeea4e07af1b65e18c06e748"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "fb400f5215dc7a52304c7fa32565806b"
server_key_iv: "329c000ddc57f44c3d10d9a3d388c16a"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.795840 9181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "f4b02bacdeea4e07af1b65e18c06e748"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "fb400f5215dc7a52304c7fa32565806b"
server_key_iv: "329c000ddc57f44c3d10d9a3d388c16a"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.797667 9181 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:16.798364 9196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:16.798540 9181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:16.798630 9181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "f4b02bacdeea4e07af1b65e18c06e748"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "fb400f5215dc7a52304c7fa32565806b"
server_key_iv: "329c000ddc57f44c3d10d9a3d388c16a"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.798704 9181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:16.809211 9181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:16.811036 9181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:16.811220 9181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:16.815337 9181 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:40825
I20260808 21:06:16.815367 9253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:40825 every 8 connection(s)
I20260808 21:06:16.815652 9181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:16.816756 9254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:16.819070 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: Bootstrap starting.
I20260808 21:06:16.819736 9254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:16.820039 9254 log.cc:824] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:16.820717 9254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: No bootstrap required, opened a new log
I20260808 21:06:16.822320 9254 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } }
I20260808 21:06:16.822484 9254 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:16.822533 9254 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b02bacdeea4e07af1b65e18c06e748, State: Initialized, Role: FOLLOWER
I20260808 21:06:16.822678 9254 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } }
I20260808 21:06:16.822763 9254 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:16.822809 9254 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:16.822857 9254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:16.823756 9254 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } }
I20260808 21:06:16.823879 9254 leader_election.cc:302] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b02bacdeea4e07af1b65e18c06e748; no voters:
I20260808 21:06:16.824055 9254 leader_election.cc:288] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:16.824294 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9181
I20260808 21:06:16.824375 9258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:16.824402 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:16.824550 9254 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:16.824616 4805 external_mini_cluster.cc:1428] Setting key d16a25783ff650781a6655890f4faa41
I20260808 21:06:16.824604 9258 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [term 1 LEADER]: Becoming Leader. State: Replica: f4b02bacdeea4e07af1b65e18c06e748, State: Running, Role: LEADER
I20260808 21:06:16.824755 9258 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } }
I20260808 21:06:16.825218 9261 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4b02bacdeea4e07af1b65e18c06e748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } } }
I20260808 21:06:16.825280 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b02bacdeea4e07af1b65e18c06e748. Latest consensus state: current_term: 1 leader_uuid: "f4b02bacdeea4e07af1b65e18c06e748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b02bacdeea4e07af1b65e18c06e748" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40825 } } }
I20260808 21:06:16.825374 9261 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:16.825485 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:16.826097 9271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:16.826862 9271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:16.828392 9271 catalog_manager.cc:1347] Generated new cluster ID: e27b3004917d42fbbdd462c97e4b298f
I20260808 21:06:16.828446 9271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:16 dist-test-slave-chsk krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:16.836989 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:16.825099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.4.177.126:40825)
0808 21:06:16.825618 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:16.825629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:16.825911 (+ 282us) server_negotiation.cc:407] Connection header received
0808 21:06:16.826099 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:16.826107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:16.826171 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:16.826345 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:16.827200 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:16.828544 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:16.829604 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:16.830247 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:16.830366 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:16.833191 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:16.833197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:16.833197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:16.833213 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:16.835306 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:16.835707 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:16.835711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:16.835711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:16.835832 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:16.836051 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:16.836052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:16.836052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:16.836372 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:16.836538 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:16.836615 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 21:06:16.836770 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":92,"threads_started":1}
I20260808 21:06:16.851524 9271 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:16.852047 9271 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:16.854539 9271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: Generated new TSK 0
I20260808 21:06:16.854739 9271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:16.889402 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:16.972237 9285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:16.972441 9285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:16.972465 9285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:16.973527 9285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:16.973562 9285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:16.973587 9285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:16.975102 9285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:16.975370 9285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:16.975627 9285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:16.980469 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:16.980486 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:16.980469 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:16.980788 9285 server_base.cc:1034] running on GCE node
I20260808 21:06:16.981043 9285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:16.981328 9285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:16.982498 9285 hybrid_clock.cc:648] HybridClock initialized: now 1786223176982482 us; error 29 us; skew 500 ppm
Aug 08 21:06:16 dist-test-slave-chsk krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223176, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:16.985140 9285 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:16.985524 9285 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:16.985599 9285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:16.985764 9285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:16.986845 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "7cf03e74df3944429c4feb55dbcf5f0d"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "7b7b53a1c00a5e0ec9526eef1e4d8f94"
server_key_iv: "494e511eac010b184f5e3839a473794e"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.987206 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "7cf03e74df3944429c4feb55dbcf5f0d"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "7b7b53a1c00a5e0ec9526eef1e4d8f94"
server_key_iv: "494e511eac010b184f5e3839a473794e"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.988478 9285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:16.989161 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:16.989409 9285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:16.989518 9285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "7cf03e74df3944429c4feb55dbcf5f0d"
format_stamp: "Formatted at 2026-08-08 21:06:16 on dist-test-slave-chsk"
server_key: "7b7b53a1c00a5e0ec9526eef1e4d8f94"
server_key_iv: "494e511eac010b184f5e3839a473794e"
server_key_version: "encryptionkey@0"
I20260808 21:06:16.989596 9285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.001436 9285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.003970 9285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.004114 9285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.004388 9285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:17.004794 9285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:17.004851 9285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.004892 9285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:17.004922 9285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.011011 9285 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:46605
I20260808 21:06:17.011059 9430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:46605 every 8 connection(s)
I20260808 21:06:17.011353 9285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:17.014966 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9285
I20260808 21:06:17.015101 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:17.015301 4805 external_mini_cluster.cc:1428] Setting key 5151798bea207424e37844c53467a5be
Aug 08 21:06:17 dist-test-slave-chsk krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223176, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:17.021828 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.012061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:60421 (local address 127.4.177.126:40825)
0808 21:06:17.012126 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.012128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.012347 (+ 219us) server_negotiation.cc:407] Connection header received
0808 21:06:17.012816 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.012817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.012830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.012894 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.013964 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.014470 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.015817 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.016116 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.016198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.018482 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.018484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.018484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:17.018493 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.020323 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.020816 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.020817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.020817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.020869 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.021172 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.021172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.021172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.021274 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.021326 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.021681 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.021761 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:06:17.022168 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.012023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40825 (local address 127.4.177.65:60421)
0808 21:06:17.012214 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.012257 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.012732 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.012957 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.012960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.013219 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:17.013860 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.013866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.014553 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.014554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.015727 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.015729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.016208 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.016209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.016343 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.016882 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:17.016890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:17.018375 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:17.020406 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.020414 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.020417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.020729 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.020933 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.020934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.020934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.021098 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.021343 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:17.021351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:17.021528 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.021660 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.021852 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260808 21:06:17.022710 9431 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40825
I20260808 21:06:17.022826 9431 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:17.022989 9431 heartbeater.cc:511] Master 127.4.177.126:40825 requested a full tablet report, sending...
I20260808 21:06:17.023582 9214 ts_manager.cc:194] Registered new tserver with Master: 7cf03e74df3944429c4feb55dbcf5f0d (127.4.177.65:46605)
I20260808 21:06:17.024262 9214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:60421
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:17.043053 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:17.125310 9437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:17.125566 9437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:17.125589 9437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:17.126739 9437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:17.126780 9437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:17.126832 9437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:17.128373 9437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:17.128609 9437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:17.128826 9437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:17.133819 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.133980 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.133819 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.133998 9437 server_base.cc:1034] running on GCE node
I20260808 21:06:17.134346 9437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:17.134604 9437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:17.135742 9437 hybrid_clock.cc:648] HybridClock initialized: now 1786223177135735 us; error 34 us; skew 500 ppm
Aug 08 21:06:17 dist-test-slave-chsk krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:17.138234 9437 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:17.138566 9437 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:17.138639 9437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:17.138792 9437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:17.139917 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "40128965ac104f518995fd7e31bda37a"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "535fe3c5972fe9427b5363243ee11e08"
server_key_iv: "1ed80959a2f3180c6ddb72264ffc3632"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.140287 9437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "40128965ac104f518995fd7e31bda37a"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "535fe3c5972fe9427b5363243ee11e08"
server_key_iv: "1ed80959a2f3180c6ddb72264ffc3632"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.141571 9437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:17.142284 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.142452 9437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:17.142552 9437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "40128965ac104f518995fd7e31bda37a"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "535fe3c5972fe9427b5363243ee11e08"
server_key_iv: "1ed80959a2f3180c6ddb72264ffc3632"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.142632 9437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.158720 9437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.161145 9437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.161329 9437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.161599 9437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:17.161933 9437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:17.161989 9437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.162045 9437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:17.162091 9437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.167682 9437 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:35077
I20260808 21:06:17.167762 9579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:35077 every 8 connection(s)
I20260808 21:06:17.168066 9437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:17 dist-test-slave-chsk krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:17.176816 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.168783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:49983 (local address 127.4.177.126:40825)
0808 21:06:17.168866 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.168869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.169057 (+ 188us) server_negotiation.cc:407] Connection header received
0808 21:06:17.169520 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.169521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.169531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.169583 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.170463 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.170958 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.172006 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.172199 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.172250 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.174268 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.174270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.174270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:17.174277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.175719 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.176080 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.176080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.176081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.176130 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.176340 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.176340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.176340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.176438 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.176488 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.176688 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.176750 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:06:17.177078 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.168732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40825 (local address 127.4.177.66:49983)
0808 21:06:17.168931 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.168968 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.169436 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.169637 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.169640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.169892 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:17.170388 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.170392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.171038 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.171038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.171931 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.171933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.172270 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.172271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.172364 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.172789 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:17.172799 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:17.174156 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:17.175796 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.175801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.175803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.176017 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.176186 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.176187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.176187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.176286 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.176491 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:17.176495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:17.176601 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.176679 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.176794 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260808 21:06:17.177654 9580 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40825
I20260808 21:06:17.177771 9580 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:17.177923 9580 heartbeater.cc:511] Master 127.4.177.126:40825 requested a full tablet report, sending...
I20260808 21:06:17.178309 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9437
I20260808 21:06:17.178357 9214 ts_manager.cc:194] Registered new tserver with Master: 40128965ac104f518995fd7e31bda37a (127.4.177.66:35077)
I20260808 21:06:17.178449 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:17.178635 4805 external_mini_cluster.cc:1428] Setting key 7975c9efbd05c3685179490e14cb3422
I20260808 21:06:17.178800 9214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:49983
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:17.203884 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:17.285454 9585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:17.285648 9585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:17.285672 9585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:17.286760 9585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:17.286795 9585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:17.286821 9585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:17.288244 9585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:17.288424 9585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:17.288596 9585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:17.293505 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.293475 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.293527 9585 server_base.cc:1034] running on GCE node
W20260808 21:06:17.293473 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.293897 9585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:17.294196 9585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:17.295380 9585 hybrid_clock.cc:648] HybridClock initialized: now 1786223177295375 us; error 37 us; skew 500 ppm
Aug 08 21:06:17 dist-test-slave-chsk krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:17.298053 9585 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:17.298468 9585 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:17.298520 9585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:17.298667 9585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:17.299738 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "58c1fa52bdc71942793abdd662a43ba6"
server_key_iv: "69b90f284008a58c5a3e30b106668ef3"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.300129 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "58c1fa52bdc71942793abdd662a43ba6"
server_key_iv: "69b90f284008a58c5a3e30b106668ef3"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.301669 9585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:17.302501 9600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.302665 9585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:17.302763 9585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "58c1fa52bdc71942793abdd662a43ba6"
server_key_iv: "69b90f284008a58c5a3e30b106668ef3"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.302819 9585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.314941 9585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.317615 9585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.317781 9585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.318038 9585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:17.318321 9585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:17.318354 9585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.318377 9585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:17.318388 9585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.324092 9585 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:43147
I20260808 21:06:17.324146 9725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:43147 every 8 connection(s)
I20260808 21:06:17.324433 9585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:17.328730 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9585
I20260808 21:06:17.328836 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:17.329056 4805 external_mini_cluster.cc:1428] Setting key 72ebd07897ed3368531097fc488e118c
Aug 08 21:06:17 dist-test-slave-chsk krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223177, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:17.333791 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.325145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:50871 (local address 127.4.177.126:40825)
0808 21:06:17.325224 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.325227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.325409 (+ 182us) server_negotiation.cc:407] Connection header received
0808 21:06:17.326025 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.326026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.326037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.326089 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.327080 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.327616 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.328563 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.328870 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.328940 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.331263 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.331264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.331264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:17.331270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.332680 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.333035 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.333036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.333036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.333086 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.333293 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.333293 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.333294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.333390 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.333440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.333659 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.333718 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:06:17.334002 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.325102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40825 (local address 127.4.177.67:50871)
0808 21:06:17.325286 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.325322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.325936 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.326141 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.326144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.326504 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:17.326997 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.327002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.327694 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.327695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.328472 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.328474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.328963 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.328964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.329066 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.329665 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:17.329674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:17.331130 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:17.332747 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.332752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.332754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.332971 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.333142 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:17.333143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:17.333143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:17.333238 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:17.333444 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:17.333448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:17.333554 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.333636 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.333758 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260808 21:06:17.334563 9726 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40825
I20260808 21:06:17.334647 9726 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:17.334758 9726 heartbeater.cc:511] Master 127.4.177.126:40825 requested a full tablet report, sending...
I20260808 21:06:17.335105 9214 ts_manager.cc:194] Registered new tserver with Master: d7f41807f3b94fd2bcb3663e2cc93ab5 (127.4.177.67:43147)
I20260808 21:06:17.335491 9214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:50871
I20260808 21:06:17.340739 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:06:17.348430 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.342047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.4.177.126:40825)
0808 21:06:17.342127 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.342129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.342229 (+ 100us) server_negotiation.cc:407] Connection header received
0808 21:06:17.342313 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.342314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.342324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.342373 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.342915 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.343476 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.344179 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.344385 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.344458 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.345701 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.345703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.345703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:17.345710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.347233 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.347594 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.347595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.347595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.347650 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:17.347867 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:17.347868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:17.347869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:17.348019 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.348127 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.348234 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.348314 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 21:06:17.350117 9214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:17.351861 9214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:17.358486 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.4.177.66:35077)
0808 21:06:17.354787 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.354792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.354806 (+ 14us) server_negotiation.cc:407] Connection header received
0808 21:06:17.354888 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.354890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.354906 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.355133 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:17.355288 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.355958 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.357565 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358056 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358147 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358210 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.358361 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.358400 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260808 21:06:17.358475 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:35077 (local address 127.0.0.1:35764)
0808 21:06:17.354755 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.354766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.354820 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.355129 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.355130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.355132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:17.355205 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.355207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.356061 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.356062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.357505 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.357507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358147 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.358227 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358241 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.358342 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.358401 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":24,"thread_start_us":47,"threads_started":1}
I20260808 21:06:17.358649 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.4.177.65:46605)
0808 21:06:17.354451 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.354456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.354467 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:17.354510 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.354512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.354529 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.354607 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:17.354974 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.356136 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.357476 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358142 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358199 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358248 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.358527 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.358569 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":140,"threads_started":1}
I20260808 21:06:17.358645 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.4.177.67:43147)
0808 21:06:17.354829 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.354834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.354844 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:17.354881 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.354883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.354898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.354971 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:17.355228 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.356266 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.357382 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358092 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358169 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358241 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.358527 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.358565 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":90,"threads_started":1}
I20260808 21:06:17.359021 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:46605 (local address 127.0.0.1:38756)
0808 21:06:17.354193 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.354253 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.354330 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.354615 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.354619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.354620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:17.354685 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.354687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.356242 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.356243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.357406 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.357408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358282 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.358390 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358393 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.358439 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.358479 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":36,"thread_start_us":82,"threads_started":1}
I20260808 21:06:17.359722 9502 tablet_service.cc:1467] Processing CreateTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd (DEFAULT_TABLE table=test-table [id=78bc52834a414558a3ad65d47abdbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:17.359728 9356 tablet_service.cc:1467] Processing CreateTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd (DEFAULT_TABLE table=test-table [id=78bc52834a414558a3ad65d47abdbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:17.360097 9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe49f619e90e42c7b05f07c34d05c0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:17.360097 9502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe49f619e90e42c7b05f07c34d05c0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:17.363300 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.354320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:43147 (local address 127.0.0.1:56522)
0808 21:06:17.354467 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.354479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.354532 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.355056 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.355057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.355059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:17.355105 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.355106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.356372 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.356373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.357292 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.357294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.358284 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.358285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.358373 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.358378 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.358434 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.358479 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260808 21:06:17.363867 9743 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Bootstrap starting.
I20260808 21:06:17.364163 9652 tablet_service.cc:1467] Processing CreateTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd (DEFAULT_TABLE table=test-table [id=78bc52834a414558a3ad65d47abdbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:17.364524 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe49f619e90e42c7b05f07c34d05c0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:17.364558 9743 tablet_bootstrap.cc:654] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:17.365244 9743 log.cc:824] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:17.365614 9742 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: Bootstrap starting.
I20260808 21:06:17.366006 9743 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: No bootstrap required, opened a new log
I20260808 21:06:17.366094 9743 ts_tablet_manager.cc:1397] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:17.366297 9742 tablet_bootstrap.cc:654] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:17.366914 9742 log.cc:824] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:17.367748 9742 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: No bootstrap required, opened a new log
I20260808 21:06:17.367836 9742 ts_tablet_manager.cc:1397] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:06:17.367791 9743 raft_consensus.cc:348] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.368003 9743 raft_consensus.cc:374] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:17.368049 9743 raft_consensus.cc:729] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40128965ac104f518995fd7e31bda37a, State: Initialized, Role: FOLLOWER
I20260808 21:06:17.368232 9746 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Bootstrap starting.
I20260808 21:06:17.368207 9743 consensus_queue.cc:260] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.368490 9743 ts_tablet_manager.cc:1428] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:17.368625 9580 heartbeater.cc:503] Master 127.4.177.126:40825 was elected leader, sending a full tablet report...
I20260808 21:06:17.368981 9746 tablet_bootstrap.cc:654] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:17.369514 9746 log.cc:824] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:17.369941 9742 raft_consensus.cc:348] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.370103 9742 raft_consensus.cc:374] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:17.370151 9742 raft_consensus.cc:729] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf03e74df3944429c4feb55dbcf5f0d, State: Initialized, Role: FOLLOWER
I20260808 21:06:17.370263 9742 consensus_queue.cc:260] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.370337 9746 tablet_bootstrap.cc:492] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: No bootstrap required, opened a new log
I20260808 21:06:17.370395 9746 ts_tablet_manager.cc:1397] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:17.370548 9431 heartbeater.cc:503] Master 127.4.177.126:40825 was elected leader, sending a full tablet report...
I20260808 21:06:17.370550 9742 ts_tablet_manager.cc:1428] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 21:06:17.372282 9746 raft_consensus.cc:348] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.372396 9746 raft_consensus.cc:374] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:17.372416 9746 raft_consensus.cc:729] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f41807f3b94fd2bcb3663e2cc93ab5, State: Initialized, Role: FOLLOWER
I20260808 21:06:17.372516 9746 consensus_queue.cc:260] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.372745 9746 ts_tablet_manager.cc:1428] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:17.372793 9726 heartbeater.cc:503] Master 127.4.177.126:40825 was elected leader, sending a full tablet report...
W20260808 21:06:17.418620 9581 tablet.cc:2369] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:17.428007 9750 raft_consensus.cc:482] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:17.428138 9750 raft_consensus.cc:504] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.428560 9750 leader_election.cc:288] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf03e74df3944429c4feb55dbcf5f0d (127.4.177.65:46605), 40128965ac104f518995fd7e31bda37a (127.4.177.66:35077)
I20260808 21:06:17.431378 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.428762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:46605 (local address 127.4.177.67:52475)
0808 21:06:17.428833 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.428848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.428909 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.429069 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.429070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.429072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:17.429132 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.429134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.429799 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.429801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.430560 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.430562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.431096 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.431097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.431192 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.431195 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.431266 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.431305 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260808 21:06:17.431378 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.428860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:52475 (local address 127.4.177.65:46605)
0808 21:06:17.428910 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.428912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.428922 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:17.428986 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.428987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.428995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.429056 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:17.429221 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.429690 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.430659 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.431016 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.431099 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.431107 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.431276 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.431313 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:17.431723 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.428952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:56113 (local address 127.4.177.66:35077)
0808 21:06:17.428996 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.428998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.429154 (+ 156us) server_negotiation.cc:407] Connection header received
0808 21:06:17.429195 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.429196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.429219 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.429272 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:17.429438 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.429906 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.430837 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.431267 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.431358 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.431366 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.431607 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.431653 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:17.431790 9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe49f619e90e42c7b05f07c34d05c0cd" candidate_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" is_pre_election: true
I20260808 21:06:17.431954 9378 raft_consensus.cc:2454] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f41807f3b94fd2bcb3663e2cc93ab5 in term 0.
I20260808 21:06:17.431986 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.428883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:35077 (local address 127.4.177.67:56113)
0808 21:06:17.429073 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.429083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.429137 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.429291 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.429292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.429294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:17.429369 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.429370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.430000 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.430002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.430748 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:17.430750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.431360 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.431361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:17.431475 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.431492 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.431833 (+ 341us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.431886 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":124,"threads_started":1}
I20260808 21:06:17.432307 9603 leader_election.cc:302] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf03e74df3944429c4feb55dbcf5f0d, d7f41807f3b94fd2bcb3663e2cc93ab5; no voters:
I20260808 21:06:17.432366 9523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe49f619e90e42c7b05f07c34d05c0cd" candidate_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40128965ac104f518995fd7e31bda37a" is_pre_election: true
I20260808 21:06:17.432444 9750 raft_consensus.cc:2781] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:17.432488 9750 raft_consensus.cc:482] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:17.432508 9750 raft_consensus.cc:3037] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:17.432508 9523 raft_consensus.cc:2454] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f41807f3b94fd2bcb3663e2cc93ab5 in term 0.
I20260808 21:06:17.433246 9750 raft_consensus.cc:504] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.433382 9750 leader_election.cc:288] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [CANDIDATE]: Term 1 election: Requested vote from peers 7cf03e74df3944429c4feb55dbcf5f0d (127.4.177.65:46605), 40128965ac104f518995fd7e31bda37a (127.4.177.66:35077)
I20260808 21:06:17.433559 9378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe49f619e90e42c7b05f07c34d05c0cd" candidate_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf03e74df3944429c4feb55dbcf5f0d"
I20260808 21:06:17.433573 9523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe49f619e90e42c7b05f07c34d05c0cd" candidate_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40128965ac104f518995fd7e31bda37a"
I20260808 21:06:17.433645 9523 raft_consensus.cc:3037] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:17.433645 9378 raft_consensus.cc:3037] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:17.434609 9378 raft_consensus.cc:2454] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f41807f3b94fd2bcb3663e2cc93ab5 in term 1.
I20260808 21:06:17.434609 9523 raft_consensus.cc:2454] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f41807f3b94fd2bcb3663e2cc93ab5 in term 1.
I20260808 21:06:17.434784 9603 leader_election.cc:302] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf03e74df3944429c4feb55dbcf5f0d, d7f41807f3b94fd2bcb3663e2cc93ab5; no voters:
I20260808 21:06:17.434895 9750 raft_consensus.cc:2781] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:17.435045 9750 raft_consensus.cc:686] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 1 LEADER]: Becoming Leader. State: Replica: d7f41807f3b94fd2bcb3663e2cc93ab5, State: Running, Role: LEADER
I20260808 21:06:17.435138 9750 consensus_queue.cc:237] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } }
I20260808 21:06:17.435994 9214 catalog_manager.cc:5495] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f41807f3b94fd2bcb3663e2cc93ab5 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f41807f3b94fd2bcb3663e2cc93ab5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 43147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:17.459952 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.453210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.4.177.67:43147)
0808 21:06:17.453282 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.453285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.453296 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:17.453337 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.453338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.453361 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.453436 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:17.453585 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.454458 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.458986 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:17.459247 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:17.459338 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:17.459534 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:17.459654 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:17.459744 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.459803 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.459826 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:17.461951 9523 raft_consensus.cc:1261] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 1 FOLLOWER]: Refusing update from remote peer d7f41807f3b94fd2bcb3663e2cc93ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:17.461983 9378 raft_consensus.cc:1261] T fe49f619e90e42c7b05f07c34d05c0cd P 7cf03e74df3944429c4feb55dbcf5f0d [term 1 FOLLOWER]: Refusing update from remote peer d7f41807f3b94fd2bcb3663e2cc93ab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:17.462366 9752 consensus_queue.cc:1035] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40128965ac104f518995fd7e31bda37a" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:17.462509 9750 consensus_queue.cc:1035] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf03e74df3944429c4feb55dbcf5f0d" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:17.465569 9756 mvcc.cc:205] Tried to move back new op lower bound from 7316370134881370112 to 7316370134775209984. Current Snapshot: MvccSnapshot[applied={T|T < 7316370134881370112}]
I20260808 21:06:17.467537 9758 mvcc.cc:205] Tried to move back new op lower bound from 7316370134881370112 to 7316370134775209984. Current Snapshot: MvccSnapshot[applied={T|T < 7316370134881370112}]
I20260808 21:06:17.468355 9213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:17.468458 9213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:17.469295 9213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4b02bacdeea4e07af1b65e18c06e748: Sending DeleteTablet for 3 replicas of tablet fe49f619e90e42c7b05f07c34d05c0cd
I20260808 21:06:17.469666 9356 tablet_service.cc:1514] Processing DeleteTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:17 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:38756
I20260808 21:06:17.469696 9502 tablet_service.cc:1514] Processing DeleteTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:17 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:35764
I20260808 21:06:17.469907 9652 tablet_service.cc:1514] Processing DeleteTablet for tablet fe49f619e90e42c7b05f07c34d05c0cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:17 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:56522
I20260808 21:06:17.470120 9769 tablet_replica.cc:330] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: stopping tablet replica
I20260808 21:06:17.470237 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9285
I20260808 21:06:17.470284 9769 raft_consensus.cc:2229] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:17.470405 9769 raft_consensus.cc:2258] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:17.470950 9769 ts_tablet_manager.cc:1905] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:17.472296 9769 ts_tablet_manager.cc:1918] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:17.472359 9769 log.cc:1196] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/fe49f619e90e42c7b05f07c34d05c0cd
I20260808 21:06:17.472560 9769 ts_tablet_manager.cc:1939] T fe49f619e90e42c7b05f07c34d05c0cd P 40128965ac104f518995fd7e31bda37a: Deleting consensus metadata
I20260808 21:06:17.472898 9199 catalog_manager.cc:4841] TS 40128965ac104f518995fd7e31bda37a (127.4.177.66:35077): tablet fe49f619e90e42c7b05f07c34d05c0cd (table test-table [id=78bc52834a414558a3ad65d47abdbbc8]) successfully deleted
I20260808 21:06:17.479297 9771 tablet_replica.cc:330] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: stopping tablet replica
I20260808 21:06:17.479426 9771 raft_consensus.cc:2229] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:17.479569 9771 raft_consensus.cc:2258] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:17.480043 9771 ts_tablet_manager.cc:1905] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 21:06:17.480515 9199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:46605 (error 108)
I20260808 21:06:17.481047 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.480898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:46605 (local address 127.0.0.1:39596)
0808 21:06:17.480964 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.481011 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:46605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260808 21:06:17.481174 9199 catalog_manager.cc:4584] TS 7cf03e74df3944429c4feb55dbcf5f0d (127.4.177.65:46605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe49f619e90e42c7b05f07c34d05c0cd: Network error: Client connection negotiation failed: client connection to 127.4.177.65:46605: connect: Connection refused (error 111)
I20260808 21:06:17.481478 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9437
I20260808 21:06:17.483299 9771 ts_tablet_manager.cc:1918] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:17.483368 9771 log.cc:1196] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/fe49f619e90e42c7b05f07c34d05c0cd
I20260808 21:06:17.483636 9771 ts_tablet_manager.cc:1939] T fe49f619e90e42c7b05f07c34d05c0cd P d7f41807f3b94fd2bcb3663e2cc93ab5: Deleting consensus metadata
I20260808 21:06:17.487462 9199 catalog_manager.cc:4841] TS d7f41807f3b94fd2bcb3663e2cc93ab5 (127.4.177.67:43147): tablet fe49f619e90e42c7b05f07c34d05c0cd (table test-table [id=78bc52834a414558a3ad65d47abdbbc8]) successfully deleted
I20260808 21:06:17.491261 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9585
I20260808 21:06:17.497164 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9181
2026-08-08T21:06:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1995 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T21:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:17Z Disabled control of system clock
I20260808 21:06:17.528191 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33647
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:33647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 21:06:17.612136 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:17.612335 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:17.612401 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:17.613564 9776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:17.613606 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:17.613619 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:17.613632 9776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:17.613644 9776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:17.614910 9776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:33647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:17.615106 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:17.615427 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:06:17.620311 9776 server_base.cc:1034] running on GCE node
W20260808 21:06:17.620285 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.620277 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.620277 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.620788 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:17.621089 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:17.622257 9776 hybrid_clock.cc:648] HybridClock initialized: now 1786223177622250 us; error 37 us; skew 500 ppm
I20260808 21:06:17.623536 9776 webserver.cc:460] Webserver started at http://127.4.177.126:44163/ using document root <none> and password file <none>
I20260808 21:06:17.623778 9776 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:17.623849 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:17.624006 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:17.625082 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "701f1c6621f649e99b0c17097f5b90d2"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "4f93311f49489b2c8c6c9e9ca6dcbb49"
server_key_iv: "a928542acd2d214bf8e309e7c68d14b9"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.625451 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "701f1c6621f649e99b0c17097f5b90d2"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "4f93311f49489b2c8c6c9e9ca6dcbb49"
server_key_iv: "a928542acd2d214bf8e309e7c68d14b9"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.626782 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:17.627584 9792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.627760 9776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.627869 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "701f1c6621f649e99b0c17097f5b90d2"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "4f93311f49489b2c8c6c9e9ca6dcbb49"
server_key_iv: "a928542acd2d214bf8e309e7c68d14b9"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.627947 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.631325 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.631608 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.631762 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.635835 9776 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:33647
I20260808 21:06:17.635866 9852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:33647 every 8 connection(s)
I20260808 21:06:17.636222 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:17.636906 9853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:17.639846 9853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: Bootstrap starting.
I20260808 21:06:17.640504 9853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:17.640830 9853 log.cc:824] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:17.641513 9853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: No bootstrap required, opened a new log
I20260808 21:06:17.642963 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9776
I20260808 21:06:17.643067 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:17.643288 4805 external_mini_cluster.cc:1428] Setting key 65b91b356362b106a646b4b68cf69163
I20260808 21:06:17.643461 9853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } }
I20260808 21:06:17.643647 9853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:17.643685 9853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f1c6621f649e99b0c17097f5b90d2, State: Initialized, Role: FOLLOWER
I20260808 21:06:17.643872 9853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } }
I20260808 21:06:17.643968 9853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:17.644014 9853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:17.644058 9853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:17.644665 9853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } }
I20260808 21:06:17.644783 9853 leader_election.cc:302] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 701f1c6621f649e99b0c17097f5b90d2; no voters:
I20260808 21:06:17.645041 9853 leader_election.cc:288] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:17.645071 9859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:17.645265 9859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [term 1 LEADER]: Becoming Leader. State: Replica: 701f1c6621f649e99b0c17097f5b90d2, State: Running, Role: LEADER
I20260808 21:06:17.645388 9853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:17.645661 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.643803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.4.177.126:33647)
0808 21:06:17.644026 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.644036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.644072 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:06:17.644188 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.644193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.644233 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.644367 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.645110 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.645113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.645114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:17.645118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.645167 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:17.645178 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.645265 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.645441 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.645502 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260808 21:06:17.645356 9859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } }
I20260808 21:06:17.647724 9860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "701f1c6621f649e99b0c17097f5b90d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } } }
I20260808 21:06:17.647953 9860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [sys.catalog]: This master's current role is: LEADER
W20260808 21:06:17.648178 9876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:17.648236 9876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:17.648224 9861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 701f1c6621f649e99b0c17097f5b90d2. Latest consensus state: current_term: 1 leader_uuid: "701f1c6621f649e99b0c17097f5b90d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "701f1c6621f649e99b0c17097f5b90d2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33647 } } }
I20260808 21:06:17.648360 9878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:17.648429 9861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:17.648607 9878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:17.650098 9878 catalog_manager.cc:1347] Generated new cluster ID: 276e0b06a5834fcab16824a1b14a5452
I20260808 21:06:17.650152 9878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:17.677579 9878 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:17.677801 9878 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:17.683568 9878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: Generated new TSK 0
I20260808 21:06:17.683805 9878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:17.685746 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--builtin_ntp_servers=127.4.177.84:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 21:06:17.774580 9884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:17.774781 9884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:17.774804 9884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:17.776036 9884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:17.776080 9884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:17.776108 9884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:17.777459 9884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:17.777693 9884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:17.778015 9884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:17.782807 9889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.782887 9890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.782788 9892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.783080 9884 server_base.cc:1034] running on GCE node
I20260808 21:06:17.783313 9884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:17.783556 9884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:17.784725 9884 hybrid_clock.cc:648] HybridClock initialized: now 1786223177784713 us; error 29 us; skew 500 ppm
I20260808 21:06:17.785907 9884 webserver.cc:460] Webserver started at http://127.4.177.65:33573/ using document root <none> and password file <none>
I20260808 21:06:17.786154 9884 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:17.786231 9884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:17.786374 9884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:17.787521 9884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "75da8e618ffe4e88ad9c86bae5c33abf"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "deeb22d32ea0d045eb098c26dd2aec6a"
server_key_iv: "924cdebdfc3bf5e724db03e10ef50d3e"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.787894 9884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "75da8e618ffe4e88ad9c86bae5c33abf"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "deeb22d32ea0d045eb098c26dd2aec6a"
server_key_iv: "924cdebdfc3bf5e724db03e10ef50d3e"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.789193 9884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:17.789949 9898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.790122 9884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:17.790208 9884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "75da8e618ffe4e88ad9c86bae5c33abf"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "deeb22d32ea0d045eb098c26dd2aec6a"
server_key_iv: "924cdebdfc3bf5e724db03e10ef50d3e"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.790268 9884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.793568 9884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.793792 9884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.793887 9884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.794035 9884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:17.794353 9884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:17.794385 9884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.794440 9884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:17.794466 9884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.800282 9884 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:41675
I20260808 21:06:17.800336 10024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:41675 every 8 connection(s)
I20260808 21:06:17.800700 9884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:17.803897 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.801383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:51627 (local address 127.4.177.126:33647)
0808 21:06:17.801432 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.801434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.801686 (+ 252us) server_negotiation.cc:407] Connection header received
0808 21:06:17.802236 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.802237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.802245 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.802316 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.803604 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.803605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.803605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:17.803610 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.803627 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:17.803632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.803708 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.803804 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.803824 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:17.804092 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.801369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33647 (local address 127.4.177.65:51627)
0808 21:06:17.801566 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.801599 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.802123 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.802399 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.802403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.802411 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:17.803443 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:17.803443 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:17.803472 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:17.803500 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:17.803704 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:17.803712 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:17.803715 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.803816 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.803877 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260808 21:06:17.804610 10025 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33647
I20260808 21:06:17.804688 10025 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:17.804857 10025 heartbeater.cc:511] Master 127.4.177.126:33647 requested a full tablet report, sending...
I20260808 21:06:17.805356 9811 ts_manager.cc:194] Registered new tserver with Master: 75da8e618ffe4e88ad9c86bae5c33abf (127.4.177.65:41675)
I20260808 21:06:17.810384 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 9884
I20260808 21:06:17.810467 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:17.810606 4805 external_mini_cluster.cc:1428] Setting key f4c108f9048afa6fc123a60cf700c640
I20260808 21:06:17.811542 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--builtin_ntp_servers=127.4.177.84:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 21:06:17.895304 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:17.895521 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:17.895543 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:17.896723 10028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:17.896765 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:17.896817 10028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:17.898185 10028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:17.898451 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:17.898759 10028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:17.903858 10035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.904059 10028 server_base.cc:1034] running on GCE node
W20260808 21:06:17.903858 10037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:17.903893 10039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:17.904390 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:17.904650 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:17.905825 10028 hybrid_clock.cc:648] HybridClock initialized: now 1786223177905803 us; error 38 us; skew 500 ppm
I20260808 21:06:17.907368 10028 webserver.cc:460] Webserver started at http://127.4.177.66:34659/ using document root <none> and password file <none>
I20260808 21:06:17.907620 10028 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:17.907692 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:17.907855 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:17.908948 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "7742b51e1f8845ddaefc1dc50be314c1"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "76834ac7952e9af788a6809264cd553c"
server_key_iv: "d76fd5992cfd0d7925097f68196a9a96"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.909327 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "7742b51e1f8845ddaefc1dc50be314c1"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "76834ac7952e9af788a6809264cd553c"
server_key_iv: "d76fd5992cfd0d7925097f68196a9a96"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.910691 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:17.911489 10045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.911664 10028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:17.911765 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "7742b51e1f8845ddaefc1dc50be314c1"
format_stamp: "Formatted at 2026-08-08 21:06:17 on dist-test-slave-chsk"
server_key: "76834ac7952e9af788a6809264cd553c"
server_key_iv: "d76fd5992cfd0d7925097f68196a9a96"
server_key_version: "encryptionkey@0"
I20260808 21:06:17.911846 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:17.915225 10028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:17.915512 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:17.915659 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:17.915880 10028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:17.916188 10028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:17.916250 10028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.916285 10028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:17.916314 10028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:17.922165 10028 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:42025
I20260808 21:06:17.922248 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:42025 every 8 connection(s)
I20260808 21:06:17.922585 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:17.925575 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.923405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:40121 (local address 127.4.177.126:33647)
0808 21:06:17.923454 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:17.923456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:17.923691 (+ 235us) server_negotiation.cc:407] Connection header received
0808 21:06:17.924136 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:17.924137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:17.924142 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:17.924209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:17.925351 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:17.925353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:17.925353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:17.925358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:17.925374 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:17.925379 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:17.925439 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:17.925507 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 21:06:17.925520 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:06:17.925768 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:17.923368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33647 (local address 127.4.177.66:40121)
0808 21:06:17.923558 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:17.923600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:17.924053 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:17.924263 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:17.924266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:17.924274 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:17.925181 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:17.925182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:17.925218 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:17.925245 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:17.925449 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:17.925454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:17.925456 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:17.925515 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 21:06:17.925566 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260808 21:06:17.926344 10176 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33647
I20260808 21:06:17.926460 10176 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:17.926635 10176 heartbeater.cc:511] Master 127.4.177.126:33647 requested a full tablet report, sending...
I20260808 21:06:17.926652 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10028
I20260808 21:06:17.926755 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:17.926961 4805 external_mini_cluster.cc:1428] Setting key 5ca960edbf04b0dda28caab84ee77f16
I20260808 21:06:17.927027 9811 ts_manager.cc:194] Registered new tserver with Master: 7742b51e1f8845ddaefc1dc50be314c1 (127.4.177.66:42025)
I20260808 21:06:17.927937 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--builtin_ntp_servers=127.4.177.84:46235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 21:06:18.009002 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:18.009224 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:18.009248 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:18.010362 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:18.010397 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:18.010423 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:18.011819 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:18.012008 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:18.012253 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:18.017261 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.017341 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.017257 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:18.017482 10179 server_base.cc:1034] running on GCE node
I20260808 21:06:18.017684 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:18.017882 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:18.019014 10179 hybrid_clock.cc:648] HybridClock initialized: now 1786223178019000 us; error 31 us; skew 500 ppm
I20260808 21:06:18.020370 10179 webserver.cc:460] Webserver started at http://127.4.177.67:39607/ using document root <none> and password file <none>
I20260808 21:06:18.020608 10179 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:18.020687 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:18.020861 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:18.021934 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "7893373e45e64df3a84c1f28b3b71e61"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "3c253333078c0a72e7a10d6e9c942cb1"
server_key_iv: "7e0f89b1972800573d5447efff0cff18"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.022323 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "7893373e45e64df3a84c1f28b3b71e61"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "3c253333078c0a72e7a10d6e9c942cb1"
server_key_iv: "7e0f89b1972800573d5447efff0cff18"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.023759 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:18.024593 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.024770 10179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:18.024847 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "7893373e45e64df3a84c1f28b3b71e61"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "3c253333078c0a72e7a10d6e9c942cb1"
server_key_iv: "7e0f89b1972800573d5447efff0cff18"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.024931 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:18.028324 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:18.028578 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:18.028714 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:18.028927 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:18.029244 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:18.029286 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.029309 10179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:18.029352 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.035130 10179 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:37079
I20260808 21:06:18.035249 10324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:37079 every 8 connection(s)
I20260808 21:06:18.035570 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:18.038602 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.036342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:49713 (local address 127.4.177.126:33647)
0808 21:06:18.036411 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.036413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.036636 (+ 223us) server_negotiation.cc:407] Connection header received
0808 21:06:18.037144 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.037145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.037153 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.037219 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.038325 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.038327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.038327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.038332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.038347 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.038351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.038413 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.038515 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.038532 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:18.038803 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.036327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33647 (local address 127.4.177.67:49713)
0808 21:06:18.036521 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.036552 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.037029 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.037292 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.037301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.037308 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.038174 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.038175 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.038204 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.038231 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.038412 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.038417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.038420 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.038512 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.038574 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260808 21:06:18.039328 10325 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33647
I20260808 21:06:18.039409 10325 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:18.039575 10325 heartbeater.cc:511] Master 127.4.177.126:33647 requested a full tablet report, sending...
I20260808 21:06:18.039935 9811 ts_manager.cc:194] Registered new tserver with Master: 7893373e45e64df3a84c1f28b3b71e61 (127.4.177.67:37079)
I20260808 21:06:18.043341 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10179
I20260808 21:06:18.043447 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:18.043622 4805 external_mini_cluster.cc:1428] Setting key 160f19192da62058cd8b2744b6be069b
I20260808 21:06:18.044438 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:06:18.046916 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.045785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.4.177.126:33647)
0808 21:06:18.045832 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.045834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.045962 (+ 128us) server_negotiation.cc:407] Connection header received
0808 21:06:18.046003 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.046003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.046007 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.046060 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.046709 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.046710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.046710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.046713 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.046725 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.046729 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.046771 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.046852 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.046872 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:06:18.048395 9811 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:18.049911 9811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:18.053882 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:37079 (local address 127.0.0.1:58076)
0808 21:06:18.052594 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.052603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.052655 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.053112 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.053113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.053114 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.053417 (+ 303us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.053418 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.053424 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.053434 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.053658 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.053696 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.053707 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.053778 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.053791 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":68,"threads_started":1}
I20260808 21:06:18.053928 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.4.177.67:37079)
0808 21:06:18.052589 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.052593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.052652 (+ 59us) server_negotiation.cc:407] Connection header received
0808 21:06:18.052808 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.052813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.052822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.052888 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.053514 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.053521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.053521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.053526 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.053561 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.053584 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.053655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.053802 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.053821 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260808 21:06:18.054070 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.4.177.65:41675)
0808 21:06:18.052326 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.052331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.052548 (+ 217us) server_negotiation.cc:407] Connection header received
0808 21:06:18.052666 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.052670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.052680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.052736 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.053724 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.053732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.053733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.053738 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.053773 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.053801 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.053835 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.053955 (+ 120us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.053973 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260808 21:06:18.054294 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41675 (local address 127.0.0.1:41384)
0808 21:06:18.052433 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.052474 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.052577 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.053092 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.053093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.053096 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.053603 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.053603 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.053607 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.053615 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.053881 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.053881 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.053882 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.054213 (+ 331us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.054228 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":108,"threads_started":1}
I20260808 21:06:18.054455 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.4.177.66:42025)
0808 21:06:18.052416 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.052420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.052635 (+ 215us) server_negotiation.cc:407] Connection header received
0808 21:06:18.052666 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.052668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.052679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.053709 (+ 1030us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.054170 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.054174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.054174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.054178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.054209 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.054221 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.054283 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.054348 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.054366 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260808 21:06:18.054492 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.052068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:42025 (local address 127.0.0.1:41394)
0808 21:06:18.052431 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.052471 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.052569 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.052755 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.052761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.052768 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.053285 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.053286 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.053303 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.053329 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.054281 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.054282 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.054283 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.054408 (+ 125us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.054421 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":64,"threads_started":1}
I20260808 21:06:18.054824 10251 tablet_service.cc:1467] Processing CreateTablet for tablet 386c119471614eea922e2b606d2a9f25 (DEFAULT_TABLE table=test-table [id=176a1d6bfe924518a4ac4accc34fb1d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:18.054848 9952 tablet_service.cc:1467] Processing CreateTablet for tablet 386c119471614eea922e2b606d2a9f25 (DEFAULT_TABLE table=test-table [id=176a1d6bfe924518a4ac4accc34fb1d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:18.055236 9952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c119471614eea922e2b606d2a9f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.055236 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c119471614eea922e2b606d2a9f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.058616 10340 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Bootstrap starting.
I20260808 21:06:18.059306 10339 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: Bootstrap starting.
I20260808 21:06:18.059376 10340 tablet_bootstrap.cc:654] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.059237 10100 tablet_service.cc:1467] Processing CreateTablet for tablet 386c119471614eea922e2b606d2a9f25 (DEFAULT_TABLE table=test-table [id=176a1d6bfe924518a4ac4accc34fb1d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:18.059604 10100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 386c119471614eea922e2b606d2a9f25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.059695 10340 log.cc:824] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.059794 10339 tablet_bootstrap.cc:654] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.060082 10339 log.cc:824] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.060534 10340 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: No bootstrap required, opened a new log
I20260808 21:06:18.060685 10340 ts_tablet_manager.cc:1397] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:18.060829 10339 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: No bootstrap required, opened a new log
I20260808 21:06:18.060892 10339 ts_tablet_manager.cc:1397] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:18.062638 10339 raft_consensus.cc:348] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.062757 10339 raft_consensus.cc:374] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.062783 10339 raft_consensus.cc:729] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75da8e618ffe4e88ad9c86bae5c33abf, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.062930 10339 consensus_queue.cc:260] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.063135 10343 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Bootstrap starting.
I20260808 21:06:18.063261 10339 ts_tablet_manager.cc:1428] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:18.063410 10025 heartbeater.cc:503] Master 127.4.177.126:33647 was elected leader, sending a full tablet report...
I20260808 21:06:18.063405 10340 raft_consensus.cc:348] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.063563 10340 raft_consensus.cc:374] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.063606 10340 raft_consensus.cc:729] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7893373e45e64df3a84c1f28b3b71e61, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.063714 10340 consensus_queue.cc:260] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.063828 10343 tablet_bootstrap.cc:654] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.063997 10325 heartbeater.cc:503] Master 127.4.177.126:33647 was elected leader, sending a full tablet report...
I20260808 21:06:18.064224 10343 log.cc:824] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.063987 10340 ts_tablet_manager.cc:1428] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:18.065016 10343 tablet_bootstrap.cc:492] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: No bootstrap required, opened a new log
I20260808 21:06:18.065111 10343 ts_tablet_manager.cc:1397] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:18.066869 10343 raft_consensus.cc:348] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.067013 10343 raft_consensus.cc:374] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.067060 10343 raft_consensus.cc:729] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7742b51e1f8845ddaefc1dc50be314c1, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.067211 10343 consensus_queue.cc:260] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.067458 10343 ts_tablet_manager.cc:1428] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260808 21:06:18.067524 10176 heartbeater.cc:503] Master 127.4.177.126:33647 was elected leader, sending a full tablet report...
I20260808 21:06:18.162608 10345 raft_consensus.cc:482] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:18.162756 10345 raft_consensus.cc:504] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.163125 10345 leader_election.cc:288] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75da8e618ffe4e88ad9c86bae5c33abf (127.4.177.65:41675), 7742b51e1f8845ddaefc1dc50be314c1 (127.4.177.66:42025)
I20260808 21:06:18.164551 10335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.163413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:55275 (local address 127.4.177.65:41675)
0808 21:06:18.163493 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.163495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.163536 (+ 41us) server_negotiation.cc:407] Connection header received
0808 21:06:18.163568 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.163570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.163575 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.163633 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.164262 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.164264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.164264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.164269 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.164293 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.164298 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.164379 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.164463 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.164479 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 21:06:18.164558 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.163342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41675 (local address 127.4.177.67:55275)
0808 21:06:18.163436 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.163448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.163491 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.163649 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.163650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.163653 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.164134 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.164134 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.164142 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.164153 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.164383 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.164383 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.164384 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.164459 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.164475 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 21:06:18.164610 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.163585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:55411 (local address 127.4.177.66:42025)
0808 21:06:18.163648 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.163650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.163659 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:18.163684 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.163685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.163689 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.163750 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.164375 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.164376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.164377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.164381 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.164396 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.164401 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.164457 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.164514 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.164526 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 21:06:18.164839 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.163360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:42025 (local address 127.4.177.67:55411)
0808 21:06:18.163538 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.163549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.163593 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.163758 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.163759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.163761 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:18.164273 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:18.164274 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:18.164279 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:18.164288 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:18.164466 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:18.164466 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:18.164467 (+ 1us) client_negotiation.cc:769] Sending connection context
0808 21:06:18.164515 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 21:06:18.164526 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":126,"threads_started":1}
I20260808 21:06:18.165019 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c119471614eea922e2b606d2a9f25" candidate_uuid: "7893373e45e64df3a84c1f28b3b71e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" is_pre_election: true
I20260808 21:06:18.165200 9972 raft_consensus.cc:2454] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7893373e45e64df3a84c1f28b3b71e61 in term 0.
I20260808 21:06:18.165202 10122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c119471614eea922e2b606d2a9f25" candidate_uuid: "7893373e45e64df3a84c1f28b3b71e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7742b51e1f8845ddaefc1dc50be314c1" is_pre_election: true
I20260808 21:06:18.165371 10122 raft_consensus.cc:2454] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7893373e45e64df3a84c1f28b3b71e61 in term 0.
I20260808 21:06:18.165455 10200 leader_election.cc:302] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75da8e618ffe4e88ad9c86bae5c33abf, 7893373e45e64df3a84c1f28b3b71e61; no voters:
I20260808 21:06:18.165593 10345 raft_consensus.cc:2781] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:18.165642 10345 raft_consensus.cc:482] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:18.165673 10345 raft_consensus.cc:3037] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.166412 10345 raft_consensus.cc:504] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.166575 10345 leader_election.cc:288] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [CANDIDATE]: Term 1 election: Requested vote from peers 75da8e618ffe4e88ad9c86bae5c33abf (127.4.177.65:41675), 7742b51e1f8845ddaefc1dc50be314c1 (127.4.177.66:42025)
I20260808 21:06:18.166725 9972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c119471614eea922e2b606d2a9f25" candidate_uuid: "7893373e45e64df3a84c1f28b3b71e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75da8e618ffe4e88ad9c86bae5c33abf"
I20260808 21:06:18.166726 10122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "386c119471614eea922e2b606d2a9f25" candidate_uuid: "7893373e45e64df3a84c1f28b3b71e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7742b51e1f8845ddaefc1dc50be314c1"
I20260808 21:06:18.166821 9972 raft_consensus.cc:3037] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.166823 10122 raft_consensus.cc:3037] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.167603 9972 raft_consensus.cc:2454] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7893373e45e64df3a84c1f28b3b71e61 in term 1.
I20260808 21:06:18.167670 10122 raft_consensus.cc:2454] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7893373e45e64df3a84c1f28b3b71e61 in term 1.
I20260808 21:06:18.167802 10200 leader_election.cc:302] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75da8e618ffe4e88ad9c86bae5c33abf, 7893373e45e64df3a84c1f28b3b71e61; no voters:
I20260808 21:06:18.167888 10345 raft_consensus.cc:2781] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:18.168040 10345 raft_consensus.cc:686] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 1 LEADER]: Becoming Leader. State: Replica: 7893373e45e64df3a84c1f28b3b71e61, State: Running, Role: LEADER
I20260808 21:06:18.168144 10345 consensus_queue.cc:237] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } }
I20260808 21:06:18.168985 9811 catalog_manager.cc:5495] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7893373e45e64df3a84c1f28b3b71e61 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "7893373e45e64df3a84c1f28b3b71e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7893373e45e64df3a84c1f28b3b71e61" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 37079 } health_report { overall_health: HEALTHY } } }
W20260808 21:06:18.173130 10177 tablet.cc:2369] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:18.200150 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.199284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.4.177.67:37079)
0808 21:06:18.199355 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.199356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.199364 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:18.199403 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.199404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.199417 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.199468 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.199983 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.199985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.199985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.199989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.200003 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.200007 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.200054 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.200081 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.200092 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:18.202023 10122 raft_consensus.cc:1261] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 1 FOLLOWER]: Refusing update from remote peer 7893373e45e64df3a84c1f28b3b71e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:18.202030 9972 raft_consensus.cc:1261] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 1 FOLLOWER]: Refusing update from remote peer 7893373e45e64df3a84c1f28b3b71e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:18.202423 10345 consensus_queue.cc:1035] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7742b51e1f8845ddaefc1dc50be314c1" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:18.202585 10349 consensus_queue.cc:1035] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75da8e618ffe4e88ad9c86bae5c33abf" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:18.205953 10354 mvcc.cc:205] Tried to move back new op lower bound from 7316370137912717312 to 7316370137777483776. Current Snapshot: MvccSnapshot[applied={T|T < 7316370137912717312}]
I20260808 21:06:18.208246 9811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:18.208356 9811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:18.209151 9811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 701f1c6621f649e99b0c17097f5b90d2: Sending DeleteTablet for 3 replicas of tablet 386c119471614eea922e2b606d2a9f25
I20260808 21:06:18.209559 10100 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c119471614eea922e2b606d2a9f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:18 UTC) from {username='slave'} at 127.0.0.1:41394
I20260808 21:06:18.209594 9952 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c119471614eea922e2b606d2a9f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:18 UTC) from {username='slave'} at 127.0.0.1:41384
I20260808 21:06:18.209872 10251 tablet_service.cc:1514] Processing DeleteTablet for tablet 386c119471614eea922e2b606d2a9f25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:18 UTC) from {username='slave'} at 127.0.0.1:58076
I20260808 21:06:18.209918 10364 tablet_replica.cc:330] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf: stopping tablet replica
I20260808 21:06:18.210103 10365 tablet_replica.cc:330] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: stopping tablet replica
I20260808 21:06:18.210121 10364 raft_consensus.cc:2229] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:18.210189 10365 raft_consensus.cc:2229] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:18.210237 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9884
I20260808 21:06:18.210328 10365 raft_consensus.cc:2258] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:18.210536 10364 raft_consensus.cc:2258] T 386c119471614eea922e2b606d2a9f25 P 75da8e618ffe4e88ad9c86bae5c33abf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:18.212458 10365 ts_tablet_manager.cc:1905] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:18.214027 10365 ts_tablet_manager.cc:1918] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:18.214118 10365 log.cc:1196] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/386c119471614eea922e2b606d2a9f25
I20260808 21:06:18.214392 10365 ts_tablet_manager.cc:1939] T 386c119471614eea922e2b606d2a9f25 P 7893373e45e64df3a84c1f28b3b71e61: Deleting consensus metadata
I20260808 21:06:18.215322 9795 catalog_manager.cc:4841] TS 7893373e45e64df3a84c1f28b3b71e61 (127.4.177.67:37079): tablet 386c119471614eea922e2b606d2a9f25 (table test-table [id=176a1d6bfe924518a4ac4accc34fb1d7]) successfully deleted
I20260808 21:06:18.219286 10363 tablet_replica.cc:330] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: stopping tablet replica
I20260808 21:06:18.219440 10363 raft_consensus.cc:2229] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:18.219592 10363 raft_consensus.cc:2258] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:18.220088 10363 ts_tablet_manager.cc:1905] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 21:06:18.221248 9795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:41675 (error 108)
I20260808 21:06:18.221637 10363 ts_tablet_manager.cc:1918] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:18.221740 10363 log.cc:1196] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/386c119471614eea922e2b606d2a9f25
I20260808 21:06:18.221813 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.221648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41675 (local address 127.0.0.1:41396)
0808 21:06:18.221717 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.221772 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:41675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260808 21:06:18.221930 9795 catalog_manager.cc:4584] TS 75da8e618ffe4e88ad9c86bae5c33abf (127.4.177.65:41675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 386c119471614eea922e2b606d2a9f25: Network error: Client connection negotiation failed: client connection to 127.4.177.65:41675: connect: Connection refused (error 111)
I20260808 21:06:18.222048 10363 ts_tablet_manager.cc:1939] T 386c119471614eea922e2b606d2a9f25 P 7742b51e1f8845ddaefc1dc50be314c1: Deleting consensus metadata
I20260808 21:06:18.222229 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10028
I20260808 21:06:18.228868 10334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.228724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:42025 (local address 127.0.0.1:41404)
0808 21:06:18.228790 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.228830 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.66:42025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 21:06:18.228863 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10179
I20260808 21:06:18.236796 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 9776
2026-08-08T21:06:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (739 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 21:06:18.258651 4805 mini_oidc.cc:152] Starting JWKS server
I20260808 21:06:18.260071 4805 webserver.cc:460] Webserver started at https://127.0.0.1:38091/ using document root <none> and password file <none>
I20260808 21:06:18.260155 4805 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 21:06:18.260296 4805 webserver.cc:460] Webserver started at http://127.0.0.1:45709/ using document root <none> and password file <none>
2026-08-08T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:18Z Disabled control of system clock
I20260808 21:06:18.269268 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35897
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:35897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/testchainca.cert with env {}
W20260808 21:06:18.351818 10373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:18.352051 10373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:18.352075 10373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:18.353231 10373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:18.353271 10373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:18.353284 10373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:18.353296 10373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:18.353308 10373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 21:06:18.353322 10373 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 21:06:18.353332 10373 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38091/jwks.json
I20260808 21:06:18.354898 10373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:35897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:18.355113 10373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:18.355374 10373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:18.360461 10382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.360486 10380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.360462 10379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:18.360692 10373 server_base.cc:1034] running on GCE node
I20260808 21:06:18.360984 10373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:18.361346 10373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:18.362510 10373 hybrid_clock.cc:648] HybridClock initialized: now 1786223178362486 us; error 38 us; skew 500 ppm
I20260808 21:06:18.363929 10373 webserver.cc:460] Webserver started at http://127.4.177.126:36591/ using document root <none> and password file <none>
I20260808 21:06:18.364208 10373 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:18.364282 10373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:18.364444 10373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:18.365552 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "b8efdb8f38a87100f1ce7b34de419b25"
server_key_iv: "399b43fbf1845984f2a44245b24d86ac"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.365927 10373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "b8efdb8f38a87100f1ce7b34de419b25"
server_key_iv: "399b43fbf1845984f2a44245b24d86ac"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.367451 10373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:18.368217 10391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.368405 10373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.368510 10373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "b8efdb8f38a87100f1ce7b34de419b25"
server_key_iv: "399b43fbf1845984f2a44245b24d86ac"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.368594 10373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:18.390069 10373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:18.390462 10373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:18.390653 10373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:18.394979 10373 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:35897
I20260808 21:06:18.395018 10444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:35897 every 8 connection(s)
I20260808 21:06:18.395445 10373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:18.396513 10445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.398869 10445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe: Bootstrap starting.
I20260808 21:06:18.399585 10445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.399842 10445 log.cc:824] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.400527 10445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe: No bootstrap required, opened a new log
I20260808 21:06:18.402127 10445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } }
I20260808 21:06:18.402273 10445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.402297 10445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8b0f97a0fc486d8ec3ccee44cdecfe, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.402448 10445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } }
I20260808 21:06:18.402534 10445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:18.402565 10445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:18.402598 10445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.403092 10445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } }
I20260808 21:06:18.403244 10445 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8b0f97a0fc486d8ec3ccee44cdecfe; no voters:
I20260808 21:06:18.403402 10445 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:18.403614 10450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:18.403680 10445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:18.403808 10450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [term 1 LEADER]: Becoming Leader. State: Replica: 5f8b0f97a0fc486d8ec3ccee44cdecfe, State: Running, Role: LEADER
I20260808 21:06:18.403911 10450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } }
I20260808 21:06:18.403952 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10373
I20260808 21:06:18.404039 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:18.404222 4805 external_mini_cluster.cc:1428] Setting key 92c5f1a512825b2adbe4511ef46bb10f
I20260808 21:06:18.404369 10449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } } }
I20260808 21:06:18.404454 10449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.404642 10452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8b0f97a0fc486d8ec3ccee44cdecfe. Latest consensus state: current_term: 1 leader_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8b0f97a0fc486d8ec3ccee44cdecfe" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35897 } } }
I20260808 21:06:18.404729 10452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.404668 10460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:18.405902 10460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:18.407543 10460 catalog_manager.cc:1347] Generated new cluster ID: 0c6db98205a54f29ac1648fe3a93f50f
I20260808 21:06:18.407600 10460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:18.410912 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.405124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.4.177.126:35897)
0808 21:06:18.405309 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.405318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.405401 (+ 83us) server_negotiation.cc:407] Connection header received
0808 21:06:18.405563 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.405570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.405652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.406107 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.406230 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.407434 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.408581 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.409185 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.409614 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.410051 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.410055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.410055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.410262 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.410301 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.410315 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.410591 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.410620 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.410694 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260808 21:06:18.412038 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:06:18.416497 10460 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:18.417179 10460 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:18.422997 10460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8b0f97a0fc486d8ec3ccee44cdecfe: Generated new TSK 0
I20260808 21:06:18.423264 10460 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 21:06:18.449316 10479 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 21:06:18.449525 10479 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 21:06:18.449841 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.434012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.4.177.126:35897)
0808 21:06:18.434083 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.434085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.434232 (+ 147us) server_negotiation.cc:407] Connection header received
0808 21:06:18.434340 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.434340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.434351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.434414 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0808 21:06:18.434595 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.435094 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.436276 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.436458 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.436510 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.436751 (+ 241us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 21:06:18.449535 (+ 12784us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 21:06:18.449776 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260808 21:06:18.449915 10470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57810: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 21:06:18.450438 10478 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:18.433950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35897 (local address 127.0.0.1:57810)
0808 21:06:18.434152 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.434164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.434241 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.434449 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.434450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.434452 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 21:06:18.434521 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.434523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.435211 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.435212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.436188 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.436191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.436532 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.436533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.436611 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.436635 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 21:06:18.449752 (+ 13117us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 21:06:18.449912 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.177.126:35897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.177.126:35897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 21:06:18.450922 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10373
2026-08-08T21:06:18Z chronyd exiting
I20260808 21:06:18.489876 4805 test_util.cc:175] -----------------------------------------------
I20260808 21:06:18.489961 4805 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786223155551886-4805-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 21:06:18.490820 4805 mini_oidc.cc:152] Starting JWKS server
I20260808 21:06:18.491580 4805 webserver.cc:460] Webserver started at https://127.0.0.1:36967/ using document root <none> and password file <none>
I20260808 21:06:18.491623 4805 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 21:06:18.491736 4805 webserver.cc:460] Webserver started at http://127.0.0.1:46421/ using document root <none> and password file <none>
2026-08-08T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:18Z Disabled control of system clock
I20260808 21:06:18.501168 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40597
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:39279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:40597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/testchainca.cert with env {}
W20260808 21:06:18.584087 10485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:18.584297 10485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:18.584321 10485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:18.585496 10485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:18.585536 10485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:18.585549 10485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:18.585562 10485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:18.585573 10485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 21:06:18.585587 10485 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 21:06:18.585603 10485 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36967/jwks.json
I20260808 21:06:18.587112 10485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:39279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:40597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:18.587378 10485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:18.587544 10485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:18.592226 10490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.592233 10491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:18.592442 10485 server_base.cc:1034] running on GCE node
W20260808 21:06:18.592233 10493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:18.592792 10485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:18.593082 10485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:18.594223 10485 hybrid_clock.cc:648] HybridClock initialized: now 1786223178594216 us; error 31 us; skew 500 ppm
I20260808 21:06:18.598649 10485 webserver.cc:460] Webserver started at http://127.4.177.126:44901/ using document root <none> and password file <none>
I20260808 21:06:18.598860 10485 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:18.598902 10485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:18.598991 10485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:18.600095 10485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "5707af9e4da746849cad4829eb18ca7f"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "fd7b2d06ac37fb71de5cd239514c8e9b"
server_key_iv: "b962f2cefebcfc323936974506d6b730"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.600404 10485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "5707af9e4da746849cad4829eb18ca7f"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "fd7b2d06ac37fb71de5cd239514c8e9b"
server_key_iv: "b962f2cefebcfc323936974506d6b730"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.601643 10485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:18.602339 10500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.602497 10485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:18.602593 10485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "5707af9e4da746849cad4829eb18ca7f"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "fd7b2d06ac37fb71de5cd239514c8e9b"
server_key_iv: "b962f2cefebcfc323936974506d6b730"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.602653 10485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:18.610195 10485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:18.610464 10485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:18.610566 10485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:18.614725 10485 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:40597
I20260808 21:06:18.614810 10555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:40597 every 8 connection(s)
I20260808 21:06:18.615203 10485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:18.615715 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10485
I20260808 21:06:18.615835 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:18.616011 4805 external_mini_cluster.cc:1428] Setting key d751072c861dd15bf476f8137b66a4b1
I20260808 21:06:18.616446 10556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.619897 10556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f: Bootstrap starting.
I20260808 21:06:18.620652 10556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.621013 10556 log.cc:824] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.621840 10556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f: No bootstrap required, opened a new log
I20260808 21:06:18.621841 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.616603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.4.177.126:40597)
0808 21:06:18.616830 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.616837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.616863 (+ 26us) server_negotiation.cc:407] Connection header received
0808 21:06:18.617280 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.617286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.617345 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.617463 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.617727 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.618713 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.619896 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.620347 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.620439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.621177 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.621181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.621181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.621370 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.621402 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.621413 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.621496 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.621576 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.621656 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260808 21:06:18.622952 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:06:18.624192 10556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } }
I20260808 21:06:18.624331 10556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.624356 10556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5707af9e4da746849cad4829eb18ca7f, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.624437 10556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } }
I20260808 21:06:18.624485 10556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:18.624505 10556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:18.624536 10556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.625053 10556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } }
I20260808 21:06:18.625139 10556 leader_election.cc:302] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5707af9e4da746849cad4829eb18ca7f; no voters:
I20260808 21:06:18.625278 10556 leader_election.cc:288] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:18.625335 10562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:18.625491 10562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [term 1 LEADER]: Becoming Leader. State: Replica: 5707af9e4da746849cad4829eb18ca7f, State: Running, Role: LEADER
I20260808 21:06:18.625543 10556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:18.625630 10562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } }
I20260808 21:06:18.626005 10564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5707af9e4da746849cad4829eb18ca7f. Latest consensus state: current_term: 1 leader_uuid: "5707af9e4da746849cad4829eb18ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } } }
I20260808 21:06:18.626004 10563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5707af9e4da746849cad4829eb18ca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5707af9e4da746849cad4829eb18ca7f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40597 } } }
I20260808 21:06:18.626178 10563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.626766 10572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:18.627487 10572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:18.626168 10564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.629065 10572 catalog_manager.cc:1347] Generated new cluster ID: 72e1d54bbaa04984babedac622d656ff
I20260808 21:06:18.629136 10572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:18.643441 10572 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:18.643975 10572 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:18.647668 10572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5707af9e4da746849cad4829eb18ca7f: Generated new TSK 0
I20260808 21:06:18.647861 10572 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 21:06:18.675964 10590 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 21:06:18.676059 10590 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 21:06:18.676369 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.661457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.4.177.126:40597)
0808 21:06:18.661524 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.661527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.661658 (+ 131us) server_negotiation.cc:407] Connection header received
0808 21:06:18.661735 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.661736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.661748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.661807 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0808 21:06:18.661956 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.662445 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.663587 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.663762 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.663812 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.663999 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 21:06:18.676184 (+ 12185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 21:06:18.676316 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20260808 21:06:18.676440 10559 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260808 21:06:18.676554 10589 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:18.661402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40597 (local address 127.0.0.1:40360)
0808 21:06:18.661583 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.661595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.661664 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.661827 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.661828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.661829 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 21:06:18.661891 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.661893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.662526 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.662526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.663509 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.663512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.663839 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.663839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.663900 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.663905 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 21:06:18.676312 (+ 12407us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 21:06:18.676449 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.177.126:40597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260808 21:06:18.676905 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10485
2026-08-08T21:06:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 21:06:18.716789 4805 mini_oidc.cc:152] Starting JWKS server
I20260808 21:06:18.717552 4805 webserver.cc:460] Webserver started at https://127.0.0.1:34279/ using document root <none> and password file <none>
I20260808 21:06:18.717624 4805 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 21:06:18.717815 4805 webserver.cc:460] Webserver started at http://127.0.0.1:42911/ using document root <none> and password file <none>
2026-08-08T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:18Z Disabled control of system clock
I20260808 21:06:18.726923 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42151
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42151
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34279/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 21:06:18.809762 10596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:18.809976 10596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:18.809998 10596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:18.811247 10596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:18.811293 10596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:18.811307 10596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:18.811321 10596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:18.811363 10596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 21:06:18.811388 10596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 21:06:18.811411 10596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34279/jwks.json
I20260808 21:06:18.812882 10596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34279/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:18.813129 10596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:18.813386 10596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:18.818106 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.818114 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:18.818128 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:18.818477 10596 server_base.cc:1034] running on GCE node
I20260808 21:06:18.818732 10596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:18.819027 10596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:18.820174 10596 hybrid_clock.cc:648] HybridClock initialized: now 1786223178820156 us; error 61 us; skew 500 ppm
I20260808 21:06:18.821520 10596 webserver.cc:460] Webserver started at http://127.4.177.126:44243/ using document root <none> and password file <none>
I20260808 21:06:18.821779 10596 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:18.821827 10596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:18.822001 10596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:18.823107 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b9ebfc37079748c89e897946e932d501"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "12e9d09d1f16f2102d1bfaf8adf32e0d"
server_key_iv: "f822ba132e967529b856f56633555c4a"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.823573 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b9ebfc37079748c89e897946e932d501"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "12e9d09d1f16f2102d1bfaf8adf32e0d"
server_key_iv: "f822ba132e967529b856f56633555c4a"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.824926 10596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:18.825667 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:18.825847 10596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:18.825940 10596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b9ebfc37079748c89e897946e932d501"
format_stamp: "Formatted at 2026-08-08 21:06:18 on dist-test-slave-chsk"
server_key: "12e9d09d1f16f2102d1bfaf8adf32e0d"
server_key_iv: "f822ba132e967529b856f56633555c4a"
server_key_version: "encryptionkey@0"
I20260808 21:06:18.826016 10596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:18.838862 10596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:18.839280 10596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:18.839443 10596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:18.843724 10596 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42151
I20260808 21:06:18.843806 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42151 every 8 connection(s)
I20260808 21:06:18.844115 10596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:18.844789 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:18.847747 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501: Bootstrap starting.
I20260808 21:06:18.848307 10672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:18.848551 10672 log.cc:824] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:18.849229 10672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501: No bootstrap required, opened a new log
I20260808 21:06:18.850783 10672 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } }
I20260808 21:06:18.850924 10672 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:18.850946 10672 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ebfc37079748c89e897946e932d501, State: Initialized, Role: FOLLOWER
I20260808 21:06:18.851050 10672 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } }
I20260808 21:06:18.851147 10672 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:18.851208 10672 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:18.851274 10672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:18.851905 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10596
I20260808 21:06:18.851933 10672 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } }
I20260808 21:06:18.852013 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:18.852041 10672 leader_election.cc:302] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ebfc37079748c89e897946e932d501; no voters:
I20260808 21:06:18.852200 4805 external_mini_cluster.cc:1428] Setting key 38c3fab7353cd83a0731d0d287d90427
I20260808 21:06:18.852231 10672 leader_election.cc:288] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:18.852285 10675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:18.852424 10675 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [term 1 LEADER]: Becoming Leader. State: Replica: b9ebfc37079748c89e897946e932d501, State: Running, Role: LEADER
I20260808 21:06:18.852492 10672 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:18.852520 10675 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } }
I20260808 21:06:18.852867 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ebfc37079748c89e897946e932d501. Latest consensus state: current_term: 1 leader_uuid: "b9ebfc37079748c89e897946e932d501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } } }
I20260808 21:06:18.852946 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.853080 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ebfc37079748c89e897946e932d501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ebfc37079748c89e897946e932d501" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42151 } } }
I20260808 21:06:18.853193 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:18.853673 10682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:18.854214 10682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:18.856745 10682 catalog_manager.cc:1347] Generated new cluster ID: 9bc497f08e174ec4984dd9ef55f7682d
I20260808 21:06:18.856812 10682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:18.858139 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.852877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.4.177.126:42151)
0808 21:06:18.853168 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.853177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.853214 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:06:18.853334 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.853341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.853404 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.853552 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:18.853764 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.855000 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.855984 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.856411 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.856548 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.857280 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:18.857284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:18.857284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:18.857493 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:18.857533 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:18.857546 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:18.857595 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:18.857846 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 21:06:18.857923 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":68,"threads_started":1}
I20260808 21:06:18.858947 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:06:18.868889 10682 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:18.869565 10682 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:18.876995 10682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ebfc37079748c89e897946e932d501: Generated new TSK 0
I20260808 21:06:18.877215 10682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:18.885978 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:18.882450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.4.177.126:42151)
0808 21:06:18.882505 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:18.882508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:18.882557 (+ 49us) server_negotiation.cc:407] Connection header received
0808 21:06:18.882652 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:18.882653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:18.882666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:18.882772 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0808 21:06:18.882920 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.883524 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.884322 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.884548 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.884624 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.884824 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 21:06:18.885771 (+ 947us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 21:06:18.885904 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260808 21:06:18.886046 10680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 21:06:18.888018 10704 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:18.882339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42151 (local address 127.0.0.1:36670)
0808 21:06:18.882499 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:18.882512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:18.882577 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:18.882776 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:18.882777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:18.882779 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0808 21:06:18.882837 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.882841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.883614 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.883615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.884229 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:18.884232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:18.884644 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:18.884645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:18.884704 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:18.884712 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 21:06:18.886132 (+ 1420us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 21:06:18.887923 (+ 1791us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.177.126:42151: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1579,"thread_start_us":92,"threads_started":1}
I20260808 21:06:18.888253 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10596
E20260808 21:06:18.898518 10705 webserver.cc:533] Webserver: sslize failed:
E20260808 21:06:18.898615 10705 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-08T21:06:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:18Z Disabled control of system clock
I20260808 21:06:18.936851 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40805
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:36339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:40805
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/testchainca.cert with env {}
W20260808 21:06:19.017704 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:19.017896 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:19.017920 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:19.019241 10710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:19.019284 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:19.019296 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:19.019330 10710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:19.019356 10710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 21:06:19.019376 10710 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 21:06:19.019404 10710 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 21:06:19.021005 10710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:40805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:19.021250 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:19.021468 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:19.026247 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:19.026260 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:19.026262 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:19.026546 10710 server_base.cc:1034] running on GCE node
I20260808 21:06:19.026818 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:19.027132 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:19.028318 10710 hybrid_clock.cc:648] HybridClock initialized: now 1786223179028305 us; error 27 us; skew 500 ppm
I20260808 21:06:19.029498 10710 webserver.cc:460] Webserver started at http://127.4.177.126:36143/ using document root <none> and password file <none>
I20260808 21:06:19.029762 10710 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:19.029832 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:19.029989 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:19.031078 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "c082788ba24e42dab6888c8bd0ecc8ba"
format_stamp: "Formatted at 2026-08-08 21:06:19 on dist-test-slave-chsk"
server_key: "acfe71355fd735c990c847654be205be"
server_key_iv: "5cfa963a4a3ce23367c2bf9a662b52b7"
server_key_version: "encryptionkey@0"
I20260808 21:06:19.031509 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "c082788ba24e42dab6888c8bd0ecc8ba"
format_stamp: "Formatted at 2026-08-08 21:06:19 on dist-test-slave-chsk"
server_key: "acfe71355fd735c990c847654be205be"
server_key_iv: "5cfa963a4a3ce23367c2bf9a662b52b7"
server_key_version: "encryptionkey@0"
I20260808 21:06:19.032793 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:19.033499 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:19.033670 10710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:19.033776 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "c082788ba24e42dab6888c8bd0ecc8ba"
format_stamp: "Formatted at 2026-08-08 21:06:19 on dist-test-slave-chsk"
server_key: "acfe71355fd735c990c847654be205be"
server_key_iv: "5cfa963a4a3ce23367c2bf9a662b52b7"
server_key_version: "encryptionkey@0"
I20260808 21:06:19.033850 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:19.052408 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:19.052747 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:19.052906 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:19.057283 10710 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:40805
I20260808 21:06:19.057320 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:40805 every 8 connection(s)
I20260808 21:06:19.057690 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:19.058640 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:19.060946 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba: Bootstrap starting.
I20260808 21:06:19.061347 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10710
I20260808 21:06:19.061427 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:19.061599 4805 external_mini_cluster.cc:1428] Setting key 86d45b1f75fd1fe3bae26d4f61c82f94
I20260808 21:06:19.061815 10782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:19.062209 10782 log.cc:824] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:19.063011 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba: No bootstrap required, opened a new log
I20260808 21:06:19.065382 10782 raft_consensus.cc:348] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } }
I20260808 21:06:19.065554 10782 raft_consensus.cc:374] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:19.065594 10782 raft_consensus.cc:729] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c082788ba24e42dab6888c8bd0ecc8ba, State: Initialized, Role: FOLLOWER
I20260808 21:06:19.065714 10782 consensus_queue.cc:260] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } }
I20260808 21:06:19.065784 10782 raft_consensus.cc:388] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:19.065814 10782 raft_consensus.cc:482] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:19.065858 10782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:19.066586 10782 raft_consensus.cc:504] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } }
I20260808 21:06:19.066705 10782 leader_election.cc:302] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c082788ba24e42dab6888c8bd0ecc8ba; no voters:
I20260808 21:06:19.066869 10782 leader_election.cc:288] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:19.066907 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:19.067019 10787 raft_consensus.cc:686] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [term 1 LEADER]: Becoming Leader. State: Replica: c082788ba24e42dab6888c8bd0ecc8ba, State: Running, Role: LEADER
I20260808 21:06:19.067134 10782 sys_catalog.cc:564] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:19.067112 10787 consensus_queue.cc:237] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } }
I20260808 21:06:19.067631 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader c082788ba24e42dab6888c8bd0ecc8ba. Latest consensus state: current_term: 1 leader_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } } }
I20260808 21:06:19.067747 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:19.067678 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c082788ba24e42dab6888c8bd0ecc8ba" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40805 } } }
I20260808 21:06:19.067898 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:19.062154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.4.177.126:40805)
0808 21:06:19.062374 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:19.062383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:19.062420 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:06:19.062976 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:19.062984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:19.063071 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:19.063247 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:19.063419 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.064819 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.065538 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.066146 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.066272 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:19.066983 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:19.066987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:19.066987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:19.067280 (+ 293us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:19.067349 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:19.067365 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:19.067423 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:19.067522 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 21:06:19.067611 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260808 21:06:19.067996 10799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:19.067898 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:19.068454 10799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:19.068877 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:06:19.072139 10799 catalog_manager.cc:1347] Generated new cluster ID: 3c8fe1cae5584b2b9a0e4fdc2274de0f
I20260808 21:06:19.072194 10799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:19.085500 10799 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:19.086223 10799 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:19.095816 10799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c082788ba24e42dab6888c8bd0ecc8ba: Generated new TSK 0
I20260808 21:06:19.096002 10799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:19.153435 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:19.103907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.4.177.126:40805)
0808 21:06:19.103954 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:19.103956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:19.104046 (+ 90us) server_negotiation.cc:407] Connection header received
0808 21:06:19.104107 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:19.104108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:19.104120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:19.104201 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0808 21:06:19.104320 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.105102 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.105844 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.106166 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.106237 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:19.106406 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 21:06:19.153153 (+ 46747us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 21:06:19.153349 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":25}
W20260808 21:06:19.153517 10785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54866: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 21:06:19.153543 10815 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:19.103818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40805 (local address 127.0.0.1:54866)
0808 21:06:19.103972 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:19.103983 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:19.104048 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:19.104208 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:19.104209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:19.104210 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0808 21:06:19.104263 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:19.104265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.105182 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.105182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:19.105766 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:19.105768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:19.106253 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:19.106254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:19.106309 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:19.106314 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 21:06:19.153305 (+ 46991us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 21:06:19.153431 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.177.126:40805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260808 21:06:19.154083 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10710
2026-08-08T21:06:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (244 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:19 dist-test-slave-chsk krb5kdc[10829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:19 dist-test-slave-chsk krb5kdc[10829](info): set up 2 sockets
Aug 08 21:06:19 dist-test-slave-chsk krb5kdc[10829](info): commencing operation
krb5kdc: starting...
W20260808 21:06:20.411620 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:20.534228 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35889
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:35889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:20.617324 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:20.617534 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:20.617558 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:20.618660 10844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:20.618703 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:20.618719 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:20.618731 10844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:20.618744 10844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:20.620255 10844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:35889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:20.620471 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:20.620640 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:20.625667 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:20.625672 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:20.625680 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:20.625896 10844 server_base.cc:1034] running on GCE node
I20260808 21:06:20.626151 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:20.626447 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:20.627589 10844 hybrid_clock.cc:648] HybridClock initialized: now 1786223180627579 us; error 30 us; skew 500 ppm
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:20.630169 10844 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:20.630654 10844 webserver.cc:460] Webserver started at http://127.4.177.126:34089/ using document root <none> and password file <none>
I20260808 21:06:20.630909 10844 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:20.630980 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:20.631134 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:20.632282 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "1671a6f6f11e4532b94748a325600735"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "620eaaa48f6ba15503ef90abe51550da"
server_key_iv: "d5aeddd192d3a3210b6fa0e5aeb0d399"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.632661 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "1671a6f6f11e4532b94748a325600735"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "620eaaa48f6ba15503ef90abe51550da"
server_key_iv: "d5aeddd192d3a3210b6fa0e5aeb0d399"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.634073 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:20.634831 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.635004 10844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:20.635113 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "1671a6f6f11e4532b94748a325600735"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "620eaaa48f6ba15503ef90abe51550da"
server_key_iv: "d5aeddd192d3a3210b6fa0e5aeb0d399"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.635262 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:20.643424 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:20.645752 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:20.645929 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:20.650290 10844 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:35889
I20260808 21:06:20.650323 10916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:35889 every 8 connection(s)
I20260808 21:06:20.650679 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:20.651615 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:20.653939 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735: Bootstrap starting.
I20260808 21:06:20.654536 10917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:20.654840 10917 log.cc:824] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:20.655591 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735: No bootstrap required, opened a new log
I20260808 21:06:20.657302 10917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } }
I20260808 21:06:20.657439 10917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:20.657464 10917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1671a6f6f11e4532b94748a325600735, State: Initialized, Role: FOLLOWER
I20260808 21:06:20.657667 10917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } }
I20260808 21:06:20.657750 10917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:20.657792 10917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:20.657845 10917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:20.658385 10917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } }
I20260808 21:06:20.658496 10917 leader_election.cc:302] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1671a6f6f11e4532b94748a325600735; no voters:
I20260808 21:06:20.658682 10917 leader_election.cc:288] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:20.658722 10920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:20.658927 10920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [term 1 LEADER]: Becoming Leader. State: Replica: 1671a6f6f11e4532b94748a325600735, State: Running, Role: LEADER
I20260808 21:06:20.658919 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10844
I20260808 21:06:20.659044 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:20.658975 10917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:20.659075 10920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } }
I20260808 21:06:20.659243 4805 external_mini_cluster.cc:1428] Setting key 4824808ea5418b7f29c5ba81cf3f7af0
I20260808 21:06:20.659425 10922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1671a6f6f11e4532b94748a325600735. Latest consensus state: current_term: 1 leader_uuid: "1671a6f6f11e4532b94748a325600735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } } }
I20260808 21:06:20.659513 10922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:20.659550 10921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1671a6f6f11e4532b94748a325600735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671a6f6f11e4532b94748a325600735" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35889 } } }
I20260808 21:06:20.659610 10921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:20.660356 10927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:20.660959 10927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:20.662616 10927 catalog_manager.cc:1347] Generated new cluster ID: ef47e44e3b824c289922a3425e9cbb10
I20260808 21:06:20.662672 10927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:20.670398 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:20.659984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.4.177.126:35889)
0808 21:06:20.660354 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:20.660364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:20.660400 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:06:20.660528 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:20.660535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:20.660611 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:20.661446 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:20.661518 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.662874 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.663696 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.664134 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.664291 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:20.667086 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:20.667090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:20.667090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:20.667101 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:20.668863 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.669239 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.669242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.669243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.669354 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.669568 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.669569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.669569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.669823 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:20.670004 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:20.670071 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 21:06:20.670217 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":85,"threads_started":1}
I20260808 21:06:20.680392 10927 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:20.681094 10927 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:20.686748 10927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1671a6f6f11e4532b94748a325600735: Generated new TSK 0
I20260808 21:06:20.686931 10927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:20.718075 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:20.799964 10945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:20.800181 10945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:20.800212 10945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:20.801401 10945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:20.801450 10945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:20.801506 10945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:20.803067 10945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:20.803364 10945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:20.803627 10945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:20.808741 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:20.808763 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:20.808772 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:20.809021 10945 server_base.cc:1034] running on GCE node
I20260808 21:06:20.809242 10945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:20.809456 10945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:20.810597 10945 hybrid_clock.cc:648] HybridClock initialized: now 1786223180810589 us; error 32 us; skew 500 ppm
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:20.813244 10945 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:20.813650 10945 webserver.cc:460] Webserver started at http://127.4.177.65:43265/ using document root <none> and password file <none>
I20260808 21:06:20.813884 10945 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:20.813927 10945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:20.814023 10945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:20.815040 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "32aa791dfc8247a2b0060c6b234faaef"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "afb8752aa24477385c65cf40367cda12"
server_key_iv: "6aeaa07fdc51eecda4f7b883352022aa"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.815438 10945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "32aa791dfc8247a2b0060c6b234faaef"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "afb8752aa24477385c65cf40367cda12"
server_key_iv: "6aeaa07fdc51eecda4f7b883352022aa"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.816883 10945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:20.817587 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.817773 10945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:20.817888 10945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "32aa791dfc8247a2b0060c6b234faaef"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "afb8752aa24477385c65cf40367cda12"
server_key_iv: "6aeaa07fdc51eecda4f7b883352022aa"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.817966 10945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:20.831851 10945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:20.833802 10945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:20.833973 10945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:20.834223 10945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:20.834547 10945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:20.834606 10945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.834648 10945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:20.834672 10945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.841009 10945 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:43813
I20260808 21:06:20.841056 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:43813 every 8 connection(s)
I20260808 21:06:20.841380 10945 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:20.843072 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 10945
I20260808 21:06:20.843214 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:20.843425 4805 external_mini_cluster.cc:1428] Setting key 85925f00886e5d12764fe56a1c56f038
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:20.851444 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:20.842117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:44321 (local address 127.4.177.126:35889)
0808 21:06:20.842172 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:20.842175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:20.842414 (+ 239us) server_negotiation.cc:407] Connection header received
0808 21:06:20.843445 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:20.843446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:20.843460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:20.843516 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:20.844413 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.844922 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.845933 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.846150 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.846246 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:20.848527 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:20.848528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:20.848529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:20.848536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:20.849974 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.850413 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.850415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.850415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.850477 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.850740 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.850741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.850742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.850859 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:20.850938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:20.851258 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 21:06:20.851352 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:20.851630 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:20.842105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35889 (local address 127.4.177.65:44321)
0808 21:06:20.842285 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:20.842322 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:20.843323 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:20.843619 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:20.843623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:20.844027 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:20.844308 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:20.844322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.845022 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.845023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:20.845836 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:20.845838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.846258 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.846260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:20.846405 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:20.846961 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:20.846969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:20.848420 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:20.850078 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:20.850086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:20.850088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:20.850317 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:20.850553 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:20.850554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:20.850554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:20.850663 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:20.850928 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:20.850936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:20.851078 (+ 142us) client_negotiation.cc:769] Sending connection context
0808 21:06:20.851232 (+ 154us) client_negotiation.cc:240] Negotiation successful
0808 21:06:20.851389 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
I20260808 21:06:20.852164 11087 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35889
I20260808 21:06:20.852258 11087 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:20.852449 11087 heartbeater.cc:511] Master 127.4.177.126:35889 requested a full tablet report, sending...
I20260808 21:06:20.853044 10877 ts_manager.cc:194] Registered new tserver with Master: 32aa791dfc8247a2b0060c6b234faaef (127.4.177.65:43813)
I20260808 21:06:20.853616 10877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:44321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:20.868320 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:20.950644 11093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:20.950903 11093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:20.950943 11093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:20.952139 11093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:20.952214 11093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:20.952268 11093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:20.953784 11093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:20.954073 11093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:20.954319 11093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:20.959422 11099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:20.959408 11102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:20.959528 11093 server_base.cc:1034] running on GCE node
W20260808 21:06:20.959408 11098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:20.959933 11093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:20.960192 11093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:20.961378 11093 hybrid_clock.cc:648] HybridClock initialized: now 1786223180961358 us; error 39 us; skew 500 ppm
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:20.964036 11093 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:20.964419 11093 webserver.cc:460] Webserver started at http://127.4.177.66:41815/ using document root <none> and password file <none>
I20260808 21:06:20.964598 11093 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:20.964639 11093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:20.964735 11093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:20.965729 11093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "670d42a291cd400889f246a1ec1fee24"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "423337564143c9955afae8b11d595800"
server_key_iv: "22ce067622d0877729261d7d15171197"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.966027 11093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "670d42a291cd400889f246a1ec1fee24"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "423337564143c9955afae8b11d595800"
server_key_iv: "22ce067622d0877729261d7d15171197"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.967327 11093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:20.968037 11110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.968218 11093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:20.968305 11093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "670d42a291cd400889f246a1ec1fee24"
format_stamp: "Formatted at 2026-08-08 21:06:20 on dist-test-slave-chsk"
server_key: "423337564143c9955afae8b11d595800"
server_key_iv: "22ce067622d0877729261d7d15171197"
server_key_version: "encryptionkey@0"
I20260808 21:06:20.968360 11093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:20.977185 11093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:20.979045 11093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:20.979233 11093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:20.979439 11093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:20.979686 11093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:20.979715 11093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.979736 11093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:20.979749 11093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:20.985497 11093 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:41827
I20260808 21:06:20.985582 11232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:41827 every 8 connection(s)
I20260808 21:06:20.985893 11093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:20 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223180, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:20.993624 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11093
I20260808 21:06:20.993757 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:20.993963 4805 external_mini_cluster.cc:1428] Setting key 68191d7c6b69e3bf70d0c29b3773722a
I20260808 21:06:20.995899 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:20.986633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:48061 (local address 127.4.177.126:35889)
0808 21:06:20.986699 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:20.986701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:20.986882 (+ 181us) server_negotiation.cc:407] Connection header received
0808 21:06:20.987924 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:20.987926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:20.987938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:20.988004 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:20.988619 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.989130 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.990143 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.990345 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.990421 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:20.992644 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:20.992645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:20.992645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:20.992652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:20.994331 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.994812 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.994813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.994814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.994882 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:20.995179 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:20.995180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:20.995180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:20.995341 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:20.995414 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:20.995668 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 21:06:20.995761 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:06:20.995909 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:20.986619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35889 (local address 127.4.177.66:48061)
0808 21:06:20.986767 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:20.986802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:20.987809 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:20.988076 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:20.988079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:20.988342 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:20.988523 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:20.988526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.989245 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.989246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:20.990036 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:20.990038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:20.990429 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:20.990430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:20.990554 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:20.991084 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:20.991093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:20.992554 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:20.994416 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:20.994423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:20.994426 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:20.994725 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:20.994961 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:20.994961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:20.994962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:20.995089 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:20.995411 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:20.995416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:20.995535 (+ 119us) client_negotiation.cc:769] Sending connection context
0808 21:06:20.995622 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 21:06:20.995740 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260808 21:06:20.996387 11234 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35889
I20260808 21:06:20.996513 11234 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:20.996716 11234 heartbeater.cc:511] Master 127.4.177.126:35889 requested a full tablet report, sending...
I20260808 21:06:20.997141 10877 ts_manager.cc:194] Registered new tserver with Master: 670d42a291cd400889f246a1ec1fee24 (127.4.177.66:41827)
I20260808 21:06:20.997540 10877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:48061
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:21.020130 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:21.103866 11239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:21.104095 11239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:21.104120 11239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:21.105227 11239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:21.105268 11239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:21.105294 11239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:21.106820 11239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:35889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:21.107029 11239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:21.107242 11239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:21.112228 11247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:21.112231 11245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:21.112231 11244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:21.112313 11239 server_base.cc:1034] running on GCE node
I20260808 21:06:21.112658 11239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:21.112943 11239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:21.114116 11239 hybrid_clock.cc:648] HybridClock initialized: now 1786223181114113 us; error 40 us; skew 500 ppm
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.116693 11239 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:21.117134 11239 webserver.cc:460] Webserver started at http://127.4.177.67:46491/ using document root <none> and password file <none>
I20260808 21:06:21.117378 11239 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:21.117451 11239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:21.117599 11239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:21.118640 11239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "5362838bfa2c4215b549626a98db5b56"
format_stamp: "Formatted at 2026-08-08 21:06:21 on dist-test-slave-chsk"
server_key: "0b45413d6b4f4367ca516894b3cde296"
server_key_iv: "b48b672fb616cbd10cf820c3eaef03dc"
server_key_version: "encryptionkey@0"
I20260808 21:06:21.119017 11239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "5362838bfa2c4215b549626a98db5b56"
format_stamp: "Formatted at 2026-08-08 21:06:21 on dist-test-slave-chsk"
server_key: "0b45413d6b4f4367ca516894b3cde296"
server_key_iv: "b48b672fb616cbd10cf820c3eaef03dc"
server_key_version: "encryptionkey@0"
I20260808 21:06:21.120401 11239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:21.121255 11255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:21.121429 11239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:21.121537 11239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "5362838bfa2c4215b549626a98db5b56"
format_stamp: "Formatted at 2026-08-08 21:06:21 on dist-test-slave-chsk"
server_key: "0b45413d6b4f4367ca516894b3cde296"
server_key_iv: "b48b672fb616cbd10cf820c3eaef03dc"
server_key_version: "encryptionkey@0"
I20260808 21:06:21.121611 11239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:21.147502 11239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:21.149451 11239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:21.149626 11239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:21.149884 11239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:21.150216 11239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:21.150278 11239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:21.150315 11239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:21.150360 11239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:21.156571 11239 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:37933
I20260808 21:06:21.156613 11378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:37933 every 8 connection(s)
I20260808 21:06:21.156976 11239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:21.166229 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:21.157745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:48045 (local address 127.4.177.126:35889)
0808 21:06:21.157828 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.157830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.158003 (+ 173us) server_negotiation.cc:407] Connection header received
0808 21:06:21.158966 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.158968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.158978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.159028 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.159760 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.160367 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.161465 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.161643 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.161696 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:21.163711 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:21.163713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:21.163713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:21.163719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:21.165119 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.165496 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.165497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.165497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.165547 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.165724 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.165724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.165725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.165821 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:21.165871 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:21.166100 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 21:06:21.166159 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 21:06:21.166435 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:21.157699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:35889 (local address 127.4.177.67:48045)
0808 21:06:21.157881 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:21.157917 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:21.158869 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:21.159087 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:21.159090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:21.159422 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:21.159669 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:21.159674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.160455 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.160456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:21.161391 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:21.161394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.161717 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.161718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:21.161821 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:21.162247 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:21.162257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:21.163624 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:21.165196 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:21.165202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:21.165203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:21.165429 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:21.165589 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:21.165589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:21.165589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:21.165670 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:21.165878 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:21.165883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:21.165989 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 21:06:21.166072 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:06:21.166186 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260808 21:06:21.166548 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11239
I20260808 21:06:21.166636 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:21.166858 4805 external_mini_cluster.cc:1428] Setting key 216f6b174165694de07b42be99e7c8bc
I20260808 21:06:21.167096 11379 heartbeater.cc:347] Connected to a master server at 127.4.177.126:35889
I20260808 21:06:21.167217 11379 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:21.167409 11379 heartbeater.cc:511] Master 127.4.177.126:35889 requested a full tablet report, sending...
I20260808 21:06:21.167871 10877 ts_manager.cc:194] Registered new tserver with Master: 5362838bfa2c4215b549626a98db5b56 (127.4.177.67:37933)
I20260808 21:06:21.168258 10877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:48045
I20260808 21:06:21.178745 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/test-admin.keytab.
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.195843 4805 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 21:06:21.197618 11386 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:21.196730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.4.177.65:43813)
0808 21:06:21.196937 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.196942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.196954 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:21.196982 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.196984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.196998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.197068 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.197502 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40922: BlockingRecv error: recv got EOF from 127.0.0.1:40922 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
W20260808 21:06:21.197878 4805 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc' of type FILE
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.198884 4805 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.199806 4805 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:06:21.207748 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:21.200324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.4.177.65:43813)
0808 21:06:21.200396 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.200398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.200408 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:21.200495 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.200497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.200516 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.200583 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.200988 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.201723 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.202479 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.202828 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.202913 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:21.205012 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:21.205021 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:21.205022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:21.205029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:21.206435 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.206823 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.206826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.206827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.206888 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.207098 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.207098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.207099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.207314 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:21.207469 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:21.207544 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 21:06:21.207627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 21:06:21.209564 11386 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:21.208976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.4.177.65:43813)
0808 21:06:21.209028 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.209030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.209078 (+ 48us) server_negotiation.cc:407] Connection header received
0808 21:06:21.209122 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.209123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.209132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.209179 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.209514 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40930: BlockingRecv error: recv got EOF from 127.0.0.1:40930 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 21:06:21.209627 4805 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc' of type FILE
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.210558 4805 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.211417 4805 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:06:21.217975 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:21.211897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.4.177.65:43813)
0808 21:06:21.211954 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.211955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.211989 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:06:21.212051 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.212052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.212059 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.212104 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.212442 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.212922 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.213551 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.213747 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.213797 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:21.215751 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:21.215752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:21.215752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:21.215758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:21.216973 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.217284 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.217285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.217285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.217335 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.217522 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.217522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.217523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.217606 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:21.217677 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:21.217775 (+ 98us) server_negotiation.cc:299] Negotiation successful
0808 21:06:21.217833 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260808 21:06:21.219630 11386 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:21.219017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.4.177.65:43813)
0808 21:06:21.219072 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.219074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.219131 (+ 57us) server_negotiation.cc:407] Connection header received
0808 21:06:21.219256 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.219257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.219272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.219319 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.219592 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40938: BlockingRecv error: recv got EOF from 127.0.0.1:40938 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260808 21:06:21.219698 4805 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786223155551886-4805-0/krb5kdc/krb5cc' of type FILE
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.220722 4805 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:21.221561 4805 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[10829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:06:21.228449 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:21.222084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.4.177.65:43813)
0808 21:06:21.222136 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:21.222138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:21.222215 (+ 77us) server_negotiation.cc:407] Connection header received
0808 21:06:21.222271 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:21.222272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:21.222279 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:21.222328 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:21.222637 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.223069 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.223829 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:21.224043 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:21.224137 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:21.226176 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:21.226177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:21.226178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:21.226183 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:21.227560 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.227869 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.227869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.227870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.227920 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:21.228084 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:21.228085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:21.228085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:21.228171 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:21.228240 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:21.228340 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 21:06:21.228393 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 21:06:21.229127 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10945
I20260808 21:06:21.236299 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11093
I20260808 21:06:21.243255 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11239
I20260808 21:06:21.248965 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 10844
2026-08-08T21:06:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[11400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[11400](info): set up 2 sockets
Aug 08 21:06:21 dist-test-slave-chsk krb5kdc[11400](info): commencing operation
krb5kdc: starting...
W20260808 21:06:22.330612 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab.
Entry for principal oryx/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab.
I20260808 21:06:22.452167 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41497
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab
--principal=oryx/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:22.533929 11414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:22.534190 11414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:22.534228 11414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:22.535386 11414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:22.535455 11414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:22.535477 11414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:22.535498 11414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:22.535529 11414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:22.537114 11414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab
--principal=oryx/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:22.537415 11414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:22.537658 11414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:22.542454 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.542484 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.542454 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:22.542778 11414 server_base.cc:1034] running on GCE node
I20260808 21:06:22.543025 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:22.543386 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:22.544551 11414 hybrid_clock.cc:648] HybridClock initialized: now 1786223182544537 us; error 42 us; skew 500 ppm
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:22.547014 11414 init.cc:377] Logged in from keytab as oryx/127.4.177.126@KRBTEST.COM (short username oryx)
I20260808 21:06:22.547500 11414 webserver.cc:460] Webserver started at http://127.4.177.126:32875/ using document root <none> and password file <none>
I20260808 21:06:22.547734 11414 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:22.547807 11414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:22.547942 11414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:22.549007 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "6fef890ae9404a288a52f781415b870e"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "76fd85a7e15384e3df24c25af913f7f0"
server_key_iv: "752a8e17be79d9ff89525ffbdced5005"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.549391 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "6fef890ae9404a288a52f781415b870e"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "76fd85a7e15384e3df24c25af913f7f0"
server_key_iv: "752a8e17be79d9ff89525ffbdced5005"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.550765 11414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:22.551571 11430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.551765 11414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:06:22.551875 11414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "6fef890ae9404a288a52f781415b870e"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "76fd85a7e15384e3df24c25af913f7f0"
server_key_iv: "752a8e17be79d9ff89525ffbdced5005"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.551951 11414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:22.578392 11414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:22.580423 11414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:22.580600 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:22.584997 11414 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:41497
I20260808 21:06:22.585037 11489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:41497 every 8 connection(s)
I20260808 21:06:22.585436 11414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:22.586433 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:22.587616 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11414
I20260808 21:06:22.587757 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:22.587944 4805 external_mini_cluster.cc:1428] Setting key 5cd7af8dcb79aec9f50ee870d339ddda
I20260808 21:06:22.589320 11490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Bootstrap starting.
I20260808 21:06:22.590119 11490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:22.590457 11490 log.cc:824] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:22.591269 11490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: No bootstrap required, opened a new log
I20260808 21:06:22.593954 11490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } }
I20260808 21:06:22.594178 11490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:22.594256 11490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fef890ae9404a288a52f781415b870e, State: Initialized, Role: FOLLOWER
I20260808 21:06:22.594406 11490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } }
I20260808 21:06:22.594534 11490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:22.594592 11490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:22.594646 11490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 0 FOLLOWER]: Advancing to term 1
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:22.595435 11490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } }
I20260808 21:06:22.595573 11490 leader_election.cc:302] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fef890ae9404a288a52f781415b870e; no voters:
I20260808 21:06:22.595779 11490 leader_election.cc:288] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:22.595809 11495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:22.595922 11495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 LEADER]: Becoming Leader. State: Replica: 6fef890ae9404a288a52f781415b870e, State: Running, Role: LEADER
I20260808 21:06:22.596017 11495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } }
I20260808 21:06:22.596071 11490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:22.596371 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } }
I20260808 21:06:22.596448 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:22.596581 11495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef890ae9404a288a52f781415b870e. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } }
I20260808 21:06:22.596652 11495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:22.597427 11501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:22.598151 11501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:22.599982 11501 catalog_manager.cc:1347] Generated new cluster ID: 5a7f215745cb4ec783165a512c6f5367
I20260808 21:06:22.600037 11501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:22.600317 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:22.588515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.4.177.126:41497)
0808 21:06:22.588714 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:22.588721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:22.588746 (+ 25us) server_negotiation.cc:407] Connection header received
0808 21:06:22.589156 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:22.589163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:22.589208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:22.589335 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:22.590181 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.591206 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.592247 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.592981 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.593122 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:22.596028 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:22.596032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:22.596032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:22.596046 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:22.598530 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.598984 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.598986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.598987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.599072 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.599326 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.599327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.599327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.599600 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:22.599776 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:22.599944 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 21:06:22.600096 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260808 21:06:22.611086 11501 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:22.611819 11501 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:22.614759 11501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Generated new TSK 0
I20260808 21:06:22.614930 11501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
I20260808 21:06:22.639995 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:22.722115 11517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:22.722316 11517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:22.722338 11517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:22.723568 11517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:22.723611 11517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:22.723639 11517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:22.725173 11517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:22.725412 11517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:22.725639 11517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:22.730747 11522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.730722 11523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.730744 11526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:22.731027 11517 server_base.cc:1034] running on GCE node
I20260808 21:06:22.731298 11517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:22.731541 11517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:22.732685 11517 hybrid_clock.cc:648] HybridClock initialized: now 1786223182732668 us; error 39 us; skew 500 ppm
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:22.735361 11517 init.cc:377] Logged in from keytab as oryx/127.4.177.65@KRBTEST.COM (short username oryx)
I20260808 21:06:22.735833 11517 webserver.cc:460] Webserver started at http://127.4.177.65:45415/ using document root <none> and password file <none>
I20260808 21:06:22.736013 11517 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:22.736053 11517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:22.736141 11517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:22.737108 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "a0a285a4fb32a0835874ef1625f74096"
server_key_iv: "1451968367d2e4b9488c7bd5e5119d46"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.737411 11517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "a0a285a4fb32a0835874ef1625f74096"
server_key_iv: "1451968367d2e4b9488c7bd5e5119d46"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.738601 11517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:22.739419 11534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.739614 11517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:22.739720 11517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "a0a285a4fb32a0835874ef1625f74096"
server_key_iv: "1451968367d2e4b9488c7bd5e5119d46"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.739801 11517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:22.750664 11517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:22.752918 11517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:22.753091 11517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:22.753310 11517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:22.753626 11517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:22.753680 11517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.753720 11517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:22.753748 11517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.759907 11517 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:45637
I20260808 21:06:22.759958 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:45637 every 8 connection(s)
I20260808 21:06:22.760250 11517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:22.765312 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11517
I20260808 21:06:22.765389 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:22.765579 4805 external_mini_cluster.cc:1428] Setting key 8a88af8ed1188aa9725ec53c0fdd6abc
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:22.770087 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:22.760954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:40727 (local address 127.4.177.126:41497)
0808 21:06:22.761005 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:22.761007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:22.761232 (+ 225us) server_negotiation.cc:407] Connection header received
0808 21:06:22.761779 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:22.761782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:22.761795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:22.761847 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:22.762795 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.763378 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.764376 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.764582 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.764659 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:22.766991 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:22.766993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:22.766994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:22.767002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:22.768674 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.769100 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.769101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.769102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.769162 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.769429 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.769430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.769430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.769549 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:22.769624 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:22.769905 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 21:06:22.769997 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:22.770287 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:22.760917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:40727)
0808 21:06:22.761105 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:22.761138 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:22.761630 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:22.761915 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:22.761919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:22.762500 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:22.762675 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:22.762680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.763476 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.763478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:22.764281 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:22.764284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.764668 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.764669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:22.764803 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:22.765268 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:22.765275 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:22.766899 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:22.768770 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:22.768779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:22.768781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:22.769009 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:22.769241 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:22.769242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:22.769242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:22.769351 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:22.769628 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:22.769636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:22.769766 (+ 130us) client_negotiation.cc:769] Sending connection context
0808 21:06:22.769868 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 21:06:22.770048 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260808 21:06:22.770776 11658 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:22.770869 11658 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:22.771065 11658 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:22.771606 11450 ts_manager.cc:194] Registered new tserver with Master: 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:22.772204 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:40727
WARNING: no policy specified for oryx/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
I20260808 21:06:22.791450 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:22.872175 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:22.872370 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:22.872393 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:22.873525 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:22.873564 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:22.873590 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:22.875056 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:22.875349 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:22.875600 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:22.880489 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.880515 11672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:22.880501 11670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:22.880709 11663 server_base.cc:1034] running on GCE node
I20260808 21:06:22.880995 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:22.881199 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:22.882359 11663 hybrid_clock.cc:648] HybridClock initialized: now 1786223182882336 us; error 35 us; skew 500 ppm
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:22.884816 11663 init.cc:377] Logged in from keytab as oryx/127.4.177.66@KRBTEST.COM (short username oryx)
I20260808 21:06:22.885226 11663 webserver.cc:460] Webserver started at http://127.4.177.66:44455/ using document root <none> and password file <none>
I20260808 21:06:22.885453 11663 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:22.885522 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:22.885672 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:22.886744 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "82f8f6eeb9db4a834cb071e03c13a99b"
server_key_iv: "dfc421ea0ecd6510c55280571a1c2151"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.887104 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "82f8f6eeb9db4a834cb071e03c13a99b"
server_key_iv: "dfc421ea0ecd6510c55280571a1c2151"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.888428 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:22.889144 11681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.889317 11663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:22.889415 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "82f8f6eeb9db4a834cb071e03c13a99b"
server_key_iv: "dfc421ea0ecd6510c55280571a1c2151"
server_key_version: "encryptionkey@0"
I20260808 21:06:22.889500 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:22.899600 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:22.901386 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:22.901548 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:22.901762 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:22.902074 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:22.902128 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.902168 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:22.902204 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:22.908178 11663 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38891
I20260808 21:06:22.908219 11808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38891 every 8 connection(s)
I20260808 21:06:22.908533 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:22 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:22.916411 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11663
I20260808 21:06:22.916541 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:22.916781 4805 external_mini_cluster.cc:1428] Setting key a8d2dcc493f160a9669a5bca163983b1
I20260808 21:06:22.917307 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:22.909259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:55931 (local address 127.4.177.126:41497)
0808 21:06:22.909302 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:22.909304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:22.909497 (+ 193us) server_negotiation.cc:407] Connection header received
0808 21:06:22.909938 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:22.909939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:22.909948 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:22.910004 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:22.910874 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.911431 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.912350 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.912529 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.912578 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:22.914666 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:22.914667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:22.914668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:22.914673 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:22.915915 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.916278 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.916279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.916280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.916345 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:22.916604 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:22.916605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:22.916606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:22.916752 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:22.916830 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:22.917158 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 21:06:22.917245 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:06:22.917541 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:22.909204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:55931)
0808 21:06:22.909371 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:22.909411 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:22.909859 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:22.910075 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:22.910079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:22.910622 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:22.910803 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:22.910807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.911508 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.911509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:22.912271 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:22.912273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:22.912598 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:22.912599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:22.912695 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:22.913192 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:22.913200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:22.914577 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:22.915987 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:22.915993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:22.915995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:22.916199 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:22.916420 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:22.916421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:22.916421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:22.916514 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:22.916838 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:22.916845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:22.917012 (+ 167us) client_negotiation.cc:769] Sending connection context
0808 21:06:22.917120 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 21:06:22.917301 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260808 21:06:22.918028 11809 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:22.918133 11809 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:22.918303 11809 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:22.918660 11450 ts_manager.cc:194] Registered new tserver with Master: e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66:38891)
I20260808 21:06:22.919189 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:55931
WARNING: no policy specified for oryx/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
I20260808 21:06:22.941601 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:23.023469 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:23.023693 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:23.023730 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:23.024868 11814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:23.024935 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:23.024981 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:23.026437 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:23.026710 11814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:23.026991 11814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.031816 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.031849 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.031961 11814 server_base.cc:1034] running on GCE node
W20260808 21:06:23.032037 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.032215 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.032481 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.033646 11814 hybrid_clock.cc:648] HybridClock initialized: now 1786223183033634 us; error 30 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.036121 11814 init.cc:377] Logged in from keytab as oryx/127.4.177.67@KRBTEST.COM (short username oryx)
I20260808 21:06:23.036533 11814 webserver.cc:460] Webserver started at http://127.4.177.67:37449/ using document root <none> and password file <none>
I20260808 21:06:23.036777 11814 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.036844 11814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.036999 11814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:23.037976 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f8dcc699dc4e234a29a60a4481be1caa"
server_key_iv: "405fbc780f1d4ce1dd49c1ad55e0829d"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.038328 11814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f8dcc699dc4e234a29a60a4481be1caa"
server_key_iv: "405fbc780f1d4ce1dd49c1ad55e0829d"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.039732 11814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:23.040462 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.040629 11814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:23.040741 11814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f8dcc699dc4e234a29a60a4481be1caa"
server_key_iv: "405fbc780f1d4ce1dd49c1ad55e0829d"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.040817 11814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.064467 11814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.066380 11814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:23.066551 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.066804 11814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:23.067116 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:23.067191 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.067229 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:23.067277 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.073285 11814 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46831
I20260808 21:06:23.073336 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46831 every 8 connection(s)
I20260808 21:06:23.073643 11814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:23.076719 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11814
I20260808 21:06:23.076836 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:23.077044 4805 external_mini_cluster.cc:1428] Setting key d2f6ecb3f6640960038c206eab943680
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:23.083123 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.074340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:50459 (local address 127.4.177.126:41497)
0808 21:06:23.074427 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.074429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.074649 (+ 220us) server_negotiation.cc:407] Connection header received
0808 21:06:23.075185 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.075186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.075208 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.075271 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.076210 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.076714 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.078367 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.078579 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.078654 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.080742 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.080744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.080744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.080751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.081973 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.082347 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.082348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.082348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.082399 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.082604 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.082604 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.082605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.082704 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.082755 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.082986 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.083059 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 21:06:23.083359 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.074338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:50459)
0808 21:06:23.074523 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.074559 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.075060 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.075348 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.075351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.075910 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.076097 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.076102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.076820 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.076821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.078278 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.078281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.078672 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.078673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.078815 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.079304 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.079312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.080659 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.082052 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.082058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.082060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.082282 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.082456 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.082457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.082457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.082549 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.082761 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.082765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.082876 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.082956 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.083072 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260808 21:06:23.083829 11954 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:23.083938 11954 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.084113 11954 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:23.084486 11450 ts_manager.cc:194] Registered new tserver with Master: 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831)
I20260808 21:06:23.084868 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:50459
I20260808 21:06:23.088780 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.177.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.125.keytab.
Entry for principal oryx/127.4.177.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.125.keytab.
I20260808 21:06:23.114307 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:44519
--webserver_interface=127.4.177.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.125.keytab
--principal=oryx/127.4.177.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41497,127.4.177.125:44519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:23.194963 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:23.195212 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:23.195259 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:23.196360 11961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:23.196424 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:23.196447 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:23.196475 11961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:23.196493 11961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:23.198011 11961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41497,127.4.177.125:44519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.125.keytab
--principal=oryx/127.4.177.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:44519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.177.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
W20260808 21:06:23.198150 11961 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.177.126:41497,127.4.177.125:44519'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 21:06:23.198308 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:23.198526 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.203133 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.203183 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.203183 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.203533 11961 server_base.cc:1034] running on GCE node
I20260808 21:06:23.203715 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.203941 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.205078 11961 hybrid_clock.cc:648] HybridClock initialized: now 1786223183205058 us; error 36 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.207615 11961 init.cc:377] Logged in from keytab as oryx/127.4.177.125@KRBTEST.COM (short username oryx)
I20260808 21:06:23.207991 11961 webserver.cc:460] Webserver started at http://127.4.177.125:35705/ using document root <none> and password file <none>
I20260808 21:06:23.208168 11961 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.208205 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.208299 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:23.209307 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/instance:
uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "5de8c9cef52cd2816d5a4562d175e3d6"
server_key_iv: "99250d5480c7aa6be8ecfe5785d4ed5c"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.209604 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal/instance:
uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "5de8c9cef52cd2816d5a4562d175e3d6"
server_key_iv: "99250d5480c7aa6be8ecfe5785d4ed5c"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.210814 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:23.211511 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.211684 11961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.211782 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "5de8c9cef52cd2816d5a4562d175e3d6"
server_key_iv: "99250d5480c7aa6be8ecfe5785d4ed5c"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.211864 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.229746 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.232111 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:23.232285 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.236343 11961 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.125:44519
I20260808 21:06:23.236393 12033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.125:44519 every 8 connection(s)
I20260808 21:06:23.236781 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
I20260808 21:06:23.237783 12034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:23.239033 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 11961
I20260808 21:06:23.239439 12034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.125@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
I20260808 21:06:23.248656 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.240189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.4.177.126:41497)
0808 21:06:23.240242 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.240244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.240354 (+ 110us) server_negotiation.cc:407] Connection header received
0808 21:06:23.240723 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.240724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.240737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.240805 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.241321 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.241869 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.242964 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.243281 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.243334 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.245672 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.245673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.245673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.245681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.247054 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.247539 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.247540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.247540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.247607 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.247865 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.247866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.247866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.248082 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.248161 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.248464 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.248548 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:23.248803 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.240080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34896)
0808 21:06:23.240235 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.240280 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.240603 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.240821 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.240823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.241064 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.241223 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.241226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.241964 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.241965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.242870 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.242874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.243353 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.243354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.243466 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.243966 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.243977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.245584 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.247159 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.247166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.247169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.247457 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.247677 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.247678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.247678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.247792 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.248156 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.248161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.248308 (+ 147us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.248427 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.248583 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260808 21:06:23.249207 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.239688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.4.177.125:44519)
0808 21:06:23.239891 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.239900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.239931 (+ 31us) server_negotiation.cc:407] Connection header received
0808 21:06:23.240602 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.240610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.240634 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.240725 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.241384 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.242321 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.243070 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.243499 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.243568 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.246293 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.246299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.246299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.246305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.247908 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.248295 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.248298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.248298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.248358 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.248588 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.248588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.248589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.248744 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.248894 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.249015 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.249097 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260808 21:06:23.249615 12034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.125@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
I20260808 21:06:23.255278 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.249947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.4.177.125:44519)
0808 21:06:23.249981 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.249983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.250009 (+ 26us) server_negotiation.cc:407] Connection header received
0808 21:06:23.250063 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.250064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.250070 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.250111 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.250294 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.250715 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.251346 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.251529 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.251581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.253175 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.253176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.253177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.253182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.254286 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.254571 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.254572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.254572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.254645 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.254802 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.254802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.254802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.254886 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.254930 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.255073 (+ 143us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.255134 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:06:23.255417 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.249908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60220)
0808 21:06:23.249960 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.249971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.250023 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.250113 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.250114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.250226 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.250254 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.250255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.250774 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.250775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.251282 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.251284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.251591 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.251592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.251637 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.252036 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.252041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.253097 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.254354 (+ 1257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.254354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.254355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.254513 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.254713 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.254714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.254714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.254762 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.254936 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.254936 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.254968 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.255057 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.255228 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":83}
I20260808 21:06:23.256675 12034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Bootstrap starting.
I20260808 21:06:23.257494 12034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:23.257829 12034 log.cc:824] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:23.258538 12034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: No bootstrap required, opened a new log
I20260808 21:06:23.259966 12034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } }
I20260808 21:06:23.260111 12034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:23.260159 12034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e3cc4a60dc142ee9018c21bee1b371c, State: Initialized, Role: FOLLOWER
I20260808 21:06:23.260293 12034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } }
I20260808 21:06:23.260527 12041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } }
I20260808 21:06:23.260564 12034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:23.260609 12041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: FOLLOWER
W20260808 21:06:23.262274 12052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:23.262333 12052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:23.264112 11961 master_runner.cc:428] Detected that this master 1e3cc4a60dc142ee9018c21bee1b371c is joining an existing cluster
I20260808 21:06:23.264178 11961 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.177.125:44519
I20260808 21:06:23.272147 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.265181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.4.177.126:41497)
0808 21:06:23.265224 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.265226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.265392 (+ 166us) server_negotiation.cc:407] Connection header received
0808 21:06:23.265641 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.265642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.265653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.265718 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.266062 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.266528 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.267704 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.268004 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.268076 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.269119 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.269121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.269121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.269128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.270764 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271338 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.271340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.271340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.271403 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271622 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.271623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.271623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.271759 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.271836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.271971 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.272050 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:06:23.272163 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.265142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34902)
0808 21:06:23.265300 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.265318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.265423 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.265728 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.265728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.265919 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.265987 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.265989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.266613 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.266614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.267616 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.267618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.268101 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.268102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.268145 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.268590 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.268595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.269007 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.270868 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.270869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.270869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.271250 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.271481 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.271483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.271544 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.271821 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.271821 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.271867 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.271937 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.272036 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":6,"thread_start_us":89,"threads_started":1}
I20260808 21:06:23.272436 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.265418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.4.177.125:44519)
0808 21:06:23.265453 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.265455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.265464 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:23.265489 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.265490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.265497 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.265549 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.265969 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.266676 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.268131 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.268423 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.268489 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.269454 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.269455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.269456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.269463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.271285 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271649 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.271650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.271650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.271692 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271882 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.271883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.271883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.272008 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.272049 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.272333 (+ 284us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.272389 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:06:23.272804 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.265234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60236)
0808 21:06:23.265391 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.265402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.265444 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.265553 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.265554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.265783 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.265885 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.265887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.266775 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.266775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.267616 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.267618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.268533 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.268534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.268590 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.268979 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.268984 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.269364 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.271357 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.271358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.271571 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.271757 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.271758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.271758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.271813 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.272109 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.272110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.272141 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.272517 (+ 376us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.272599 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.126@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
I20260808 21:06:23.281944 12038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.274726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.4.177.125:44519)
0808 21:06:23.274773 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.274775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.274882 (+ 107us) server_negotiation.cc:407] Connection header received
0808 21:06:23.274959 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.274959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.274967 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.275008 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.275514 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.275926 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.276992 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.277210 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.277281 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.279337 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.279338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.279339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.279346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.280659 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.281113 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.281115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.281115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.281172 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.281415 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.281416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.281417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.281520 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.281598 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.281783 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.281857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:06:23.281960 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.274614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60248)
0808 21:06:23.274766 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.274812 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.274887 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.275023 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.275026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.275379 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.275433 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.275435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.276000 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.276001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.276916 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.276918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.277291 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.277292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.277376 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.277776 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.277781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.279247 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.280763 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.280772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.280774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.281018 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.281247 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.281248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.281248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.281348 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.281591 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.281598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.281650 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.281749 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.281865 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260808 21:06:23.282501 11450 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.177.125:44519: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
member_type: NON_VOTER
last_known_addr {
host: "127.4.177.125"
port: 44519
}
attrs {
promote: true
}
}
dest_uuid: "6fef890ae9404a288a52f781415b870e"
cas_config_opid_index: -1
I20260808 21:06:23.282840 11450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } }
I20260808 21:06:23.283366 12066 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.283727 12006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:23.284548 12066 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.177.125:44519
I20260808 21:06:23.284696 12006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 21:06:23.284885 11961 master.cc:537] Master@127.4.177.125:44519 shutting down...
I20260808 21:06:23.285022 12066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 21:06:23.285696 11433 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260808 21:06:23.286104 11433 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e -> Peer 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125:44519): Couldn't send request to peer 1e3cc4a60dc142ee9018c21bee1b371c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 21:06:23.286304 12064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.286386 12064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:23.287914 11961 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:23.288003 11961 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:23.288056 11961 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: stopping tablet replica
I20260808 21:06:23.301146 11961 master.cc:552] Master@127.4.177.125:44519 shutdown complete.
I20260808 21:06:23.301216 11961 master_runner.cc:305] Clearing existing system tablet
I20260808 21:06:23.301350 11961 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.301414 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.302047 11961 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260808 21:06:23.302590 12071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.302700 11961 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.302771 11961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:23.302799 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "5de8c9cef52cd2816d5a4562d175e3d6"
server_key_iv: "99250d5480c7aa6be8ecfe5785d4ed5c"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.303736 11961 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:23.305122 11961 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 21:06:23.305181 11961 log.cc:1196] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 21:06:23.305385 11961 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Deleting consensus metadata
I20260808 21:06:23.305528 11961 master_runner.cc:315] Copying system tablet from 127.4.177.126:41497
I20260808 21:06:23.306171 11961 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: tablet copy: Beginning tablet copy session from remote peer at address 127.4.177.126:41497
I20260808 21:06:23.312791 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.306551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34908)
0808 21:06:23.306709 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.306718 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.306782 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.306920 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.306921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.307075 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.307183 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.307185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.308150 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.308151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.308691 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.308692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.309153 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.309154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.309202 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.309706 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.309711 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.310002 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.311834 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.311834 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.311835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.312002 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.312208 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.312208 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.312208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.312266 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.312529 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.312530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.312566 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.312628 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.312709 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260808 21:06:23.312855 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.306604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.4.177.126:41497)
0808 21:06:23.306652 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.306654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.306797 (+ 143us) server_negotiation.cc:407] Connection header received
0808 21:06:23.306833 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.306834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.306846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.306918 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.307262 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.308069 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.308773 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.309076 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.309149 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.310092 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.310093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.310094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.310102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.311755 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.312076 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.312077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.312078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.312141 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.312327 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.312328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.312328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.312464 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.312535 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.312677 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.312757 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:23.313323 11472 tablet_copy_service.cc:140] P 6fef890ae9404a288a52f781415b870e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1e3cc4a60dc142ee9018c21bee1b371c ({username='oryx', principal='oryx/127.4.177.125@KRBTEST.COM'} at 127.0.0.1:34908)
I20260808 21:06:23.313467 11472 tablet_copy_service.cc:161] P 6fef890ae9404a288a52f781415b870e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1e3cc4a60dc142ee9018c21bee1b371c at {username='oryx', principal='oryx/127.4.177.125@KRBTEST.COM'} at 127.0.0.1:34908: session id = 1e3cc4a60dc142ee9018c21bee1b371c-00000000000000000000000000000000
I20260808 21:06:23.314232 11472 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 21:06:23.314916 11961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 21:06:23.316020 11961 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: tablet copy: Starting download of 0 data blocks...
I20260808 21:06:23.316197 11961 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: tablet copy: Starting download of 1 WAL segments...
I20260808 21:06:23.317209 11961 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 21:06:23.318020 11472 tablet_copy_service.cc:342] P 6fef890ae9404a288a52f781415b870e: Request end of tablet copy session 1e3cc4a60dc142ee9018c21bee1b371c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.177.125@KRBTEST.COM'} at 127.0.0.1:34908
I20260808 21:06:23.318089 11472 tablet_copy_service.cc:433] P 6fef890ae9404a288a52f781415b870e: ending tablet copy session 1e3cc4a60dc142ee9018c21bee1b371c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1e3cc4a60dc142ee9018c21bee1b371c
I20260808 21:06:23.320673 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.321817 12080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.321832 12081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.324147 12083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.324270 11961 server_base.cc:1034] running on GCE node
I20260808 21:06:23.324353 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.324487 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.325621 11961 hybrid_clock.cc:648] HybridClock initialized: now 1786223183325612 us; error 40 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.326982 11961 init.cc:377] Logged in from keytab as oryx/127.4.177.125@KRBTEST.COM (short username oryx)
I20260808 21:06:23.327324 11961 webserver.cc:460] Webserver started at http://127.4.177.125:34697/ using document root <none> and password file <none>
I20260808 21:06:23.327497 11961 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.327548 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.328261 11961 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:23.328845 12091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.329008 11961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:23.329078 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "5de8c9cef52cd2816d5a4562d175e3d6"
server_key_iv: "99250d5480c7aa6be8ecfe5785d4ed5c"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.329139 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.340938 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.342768 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.346711 11961 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.125:44519
I20260808 21:06:23.346978 12150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.125:44519 every 8 connection(s)
I20260808 21:06:23.347093 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
I20260808 21:06:23.348093 12151 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:06:23.348657 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Bootstrap starting.
I20260808 21:06:23.351841 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:06:23.352100 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: Bootstrap complete.
I20260808 21:06:23.352334 12151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } }
I20260808 21:06:23.352442 12151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e3cc4a60dc142ee9018c21bee1b371c, State: Initialized, Role: LEARNER
I20260808 21:06:23.352479 12151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } }
I20260808 21:06:23.352649 12151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:23.352695 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.352792 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: LEARNER
I20260808 21:06:23.354308 12166 catalog_manager.cc:1259] Loaded cluster ID: 5a7f215745cb4ec783165a512c6f5367
I20260808 21:06:23.354463 12166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: loading cluster ID for follower catalog manager: success
I20260808 21:06:23.355546 12166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: acquiring CA information for follower catalog manager: success
I20260808 21:06:23.355870 12166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:06:23.358942 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.350873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.4.177.125:44519)
0808 21:06:23.351017 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.351019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.351025 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:06:23.351045 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.351046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.351054 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.351129 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.351701 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.352557 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.353473 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.353789 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.353852 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.355564 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.355565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.355565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.355572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.357750 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.358182 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.358183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.358183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.358237 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.358466 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.358466 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.358467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.358565 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.358639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.358808 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.358879 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.4.177.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.124.keytab.
Entry for principal oryx/127.4.177.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.124.keytab.
I20260808 21:06:23.385841 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:45067
--webserver_interface=127.4.177.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.124.keytab
--principal=oryx/127.4.177.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:23.472378 12169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:23.472613 12169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:23.472649 12169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:23.473775 12169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:23.473841 12169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:23.473869 12169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:23.473901 12169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:23.473925 12169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:23.475478 12169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.124.keytab
--principal=oryx/127.4.177.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:45067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.177.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:23.475759 12169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:23.475986 12169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.480955 12177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.480957 12174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.481146 12175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.481163 12169 server_base.cc:1034] running on GCE node
I20260808 21:06:23.481505 12169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.481796 12169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.482965 12169 hybrid_clock.cc:648] HybridClock initialized: now 1786223183482948 us; error 48 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.485631 12169 init.cc:377] Logged in from keytab as oryx/127.4.177.124@KRBTEST.COM (short username oryx)
I20260808 21:06:23.486073 12169 webserver.cc:460] Webserver started at http://127.4.177.124:46643/ using document root <none> and password file <none>
I20260808 21:06:23.486320 12169 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.486390 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.486543 12169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:23.487749 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/instance:
uuid: "16422c219bfd4e4282697aa1213e3122"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f36d4ea769bab015aeda382b98417ff7"
server_key_iv: "fd853c372a003747f30ee4c9c78da769"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.488148 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal/instance:
uuid: "16422c219bfd4e4282697aa1213e3122"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f36d4ea769bab015aeda382b98417ff7"
server_key_iv: "fd853c372a003747f30ee4c9c78da769"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.489478 12169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 21:06:23.490219 12184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.490393 12169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:23.490512 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "16422c219bfd4e4282697aa1213e3122"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f36d4ea769bab015aeda382b98417ff7"
server_key_iv: "fd853c372a003747f30ee4c9c78da769"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.490597 12169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.499670 12169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.501753 12169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:23.501917 12169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.506446 12169 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.124:45067
I20260808 21:06:23.506492 12243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.124:45067 every 8 connection(s)
I20260808 21:06:23.506873 12169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
I20260808 21:06:23.507897 12244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:23.509449 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:06:23.511744 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 12169
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.124@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
I20260808 21:06:23.519326 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.510359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34920 (local address 127.4.177.126:41497)
0808 21:06:23.510413 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.510415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.510459 (+ 44us) server_negotiation.cc:407] Connection header received
0808 21:06:23.511075 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.511076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.511086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.511190 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.511961 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.512528 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.513511 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.513705 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.513785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.516337 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.516338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.516339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.516346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.517716 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.518141 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.518142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.518142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.518204 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.518514 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.518515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.518516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.518669 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.518743 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.519098 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.519220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:06:23.519505 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.510148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34920)
0808 21:06:23.510348 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.510384 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.510948 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.511271 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.511274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.511525 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.511864 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.511870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.512630 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.512631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.513413 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.513415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.513811 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.513813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.513924 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.514427 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.514437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.516254 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.517799 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.517807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.517810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.518052 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.518279 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.518280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.518280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.518442 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.518760 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.518767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.518940 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.519066 (+ 126us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.519265 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260808 21:06:23.520051 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:06:23.521579 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.512304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.4.177.124:45067)
0808 21:06:23.512478 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.512484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.512580 (+ 96us) server_negotiation.cc:407] Connection header received
0808 21:06:23.512642 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.512645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.512681 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.512765 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.513368 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.514073 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.515129 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.515613 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.515689 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.518346 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.518352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.518352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.518359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.520369 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.520772 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.520775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.520775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.520828 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.521042 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.521042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.521042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.521216 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.521366 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.521406 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.521479 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":144,"thread_start_us":119,"threads_started":1}
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.124@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
I20260808 21:06:23.526676 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.520339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60266)
0808 21:06:23.520387 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.520398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.520440 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.520618 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.520619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.520770 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.520812 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.520814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.521700 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.521701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.522411 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.522413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.522790 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.522790 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.522849 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.523265 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.523269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.524325 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.525660 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.525661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.525661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.525828 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.526035 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.526035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.526036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.526108 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.526362 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.526363 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.526405 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.526491 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.526587 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:06:23.526705 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.520437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.4.177.125:44519)
0808 21:06:23.520486 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.520488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.520496 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:23.520527 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.520527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.520536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.520584 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.520891 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.521633 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.522496 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.522719 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.522793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.524412 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.524413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.524414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.524421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.525576 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.525902 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.525903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.525904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.525963 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.526178 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.526179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.526179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.526290 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.526361 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.526520 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.526613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:06:23.527279 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.124@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
I20260808 21:06:23.533154 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.527601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.0.0.1:54306)
0808 21:06:23.527634 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.527644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.527681 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.527840 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.527841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.527995 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.528032 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.528033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.528661 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.528662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.529107 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.529109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.529505 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.529506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.529541 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.529956 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.529961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.531081 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.532398 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.532398 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.532399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.532555 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.532722 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.532723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.532723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.532773 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.532943 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.532944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.532974 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.533023 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.533093 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260808 21:06:23.533217 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.527662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.4.177.124:45067)
0808 21:06:23.527746 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.527748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.527754 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:06:23.527770 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.527771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.527778 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.527825 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.528106 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.528556 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.529174 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.529429 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.529481 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.531206 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.531207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.531208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.531214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.532325 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.532617 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.532617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.532618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.532665 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.532811 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.532812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.532812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.532894 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.532937 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.533042 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.533136 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":46}
I20260808 21:06:23.534626 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Bootstrap starting.
I20260808 21:06:23.535548 12244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:23.535951 12244 log.cc:824] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:23.536796 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: No bootstrap required, opened a new log
I20260808 21:06:23.538407 12244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } }
I20260808 21:06:23.538527 12244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:23.538550 12244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16422c219bfd4e4282697aa1213e3122, State: Initialized, Role: FOLLOWER
I20260808 21:06:23.538684 12244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } }
I20260808 21:06:23.538954 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } } }
I20260808 21:06:23.539017 12244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:23.539041 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 21:06:23.541301 12262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:23.541365 12262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:23.543937 12169 master_runner.cc:428] Detected that this master 16422c219bfd4e4282697aa1213e3122 is joining an existing cluster
I20260808 21:06:23.543989 12169 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.177.124:45067
I20260808 21:06:23.551904 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.4.177.125:44519)
0808 21:06:23.545495 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.545497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.545707 (+ 210us) server_negotiation.cc:407] Connection header received
0808 21:06:23.545781 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.545782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.545792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.545849 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.546242 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.546729 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.547710 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.548019 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.548099 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.549071 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.549073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.549073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.549080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.550653 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.550978 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.550979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.550979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.551041 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.551319 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.551320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.551321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.551469 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.551514 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.551723 (+ 209us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.551802 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:06:23.551932 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60276)
0808 21:06:23.545637 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.545646 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.545720 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.545855 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.545855 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.546106 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.546165 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.546166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.546807 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.546808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.547625 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.547627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.548107 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.548108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.548166 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.548658 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.548661 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.548989 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.550742 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.550743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.550743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.550910 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.551108 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.551109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.551109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.551226 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.551567 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.551567 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.551601 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.551684 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.551804 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":87,"threads_started":1}
I20260808 21:06:23.557822 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.4.177.124:45067)
0808 21:06:23.545581 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.545582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.551297 (+ 5715us) server_negotiation.cc:407] Connection header received
0808 21:06:23.551395 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.551396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.551406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.551460 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.551928 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.552620 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.554254 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.554427 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.554480 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.555471 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.555472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.555472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.555478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.556785 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.557149 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.557149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.557150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.557196 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.557379 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.557379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.557379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.557490 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.557540 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.557701 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.557764 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:06:23.557942 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.0.0.1:54308)
0808 21:06:23.551239 (+ 5764us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.551251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.551319 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.551464 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.551465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.551787 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.551842 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.551844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.552730 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.552731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.554157 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.554160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.554484 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.554485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.554562 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.554954 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.554959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.555396 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.556859 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.556860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.556861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.557074 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.557255 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.557256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.557256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.557320 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.557553 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.557554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.557608 (+ 54us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.557685 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.557834 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5743,"mutex_wait_us":50,"thread_start_us":81,"threads_started":1}
I20260808 21:06:23.566459 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.4.177.126:41497)
0808 21:06:23.545363 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.545365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.545780 (+ 415us) server_negotiation.cc:407] Connection header received
0808 21:06:23.545879 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.545880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.545894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.545926 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.546431 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.547206 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.548049 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.548350 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.548438 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.563963 (+ 15525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.563965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.563966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.563973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.565322 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.565682 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.565683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.565683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.565745 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.565936 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.565937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.565937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.566060 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.566141 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.566285 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.566365 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:23.566473 12270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.545262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34924)
0808 21:06:23.545713 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.545724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.545806 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.545990 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.545991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.546253 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.546333 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.546335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.547298 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.547299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.547961 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.547963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.548440 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.548440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.548538 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.563578 (+ 15040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.563582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.563859 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.565427 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.565428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.565428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.565601 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.565816 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.565816 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.565817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.565875 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.566131 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.566132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.566176 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.566260 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.566365 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"spinlock_wait_cycles":9984,"thread_start_us":98,"threads_started":1}
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223182, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.126@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
I20260808 21:06:23.572993 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11517
I20260808 21:06:23.583571 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:45637
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:41497
--webserver_port=45415
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:06:23.584719 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.568349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.4.177.124:45067)
0808 21:06:23.568384 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.568386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.568440 (+ 54us) server_negotiation.cc:407] Connection header received
0808 21:06:23.568520 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.568521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.568529 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.568573 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.568839 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.569291 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.569964 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.570146 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.570213 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.571991 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.571992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.571992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.571999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.583293 (+ 11294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.583837 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.583838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.583839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.583908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.584132 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.584133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.584134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.584287 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.584360 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.584535 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.584623 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 21:06:23.584781 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.568294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.0.0.1:54316)
0808 21:06:23.568395 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.568406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.568456 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.568576 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.568577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.568722 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.568759 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.568760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.569367 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.569368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.569873 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.569874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.570224 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.570225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.570291 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.570684 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.570688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.571908 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.583382 (+ 11474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.583383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.583384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.583631 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.583979 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.583980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.583980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.584065 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.584366 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.584367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.584422 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.584496 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.584626 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 21:06:23.585734 11450 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.177.124:45067: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "16422c219bfd4e4282697aa1213e3122"
member_type: NON_VOTER
last_known_addr {
host: "127.4.177.124"
port: 45067
}
attrs {
promote: true
}
}
dest_uuid: "6fef890ae9404a288a52f781415b870e"
cas_config_opid_index: 5
I20260808 21:06:23.585999 11450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
I20260808 21:06:23.586366 12064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 16422c219bfd4e4282697aa1213e3122 (127.4.177.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.586841 12214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:23.587400 12064 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.177.124:45067
I20260808 21:06:23.587472 12064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.587533 12064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:23.587838 12214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 21:06:23.588104 12066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:23.589461 12214 raft_consensus.cc:858] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
W20260808 21:06:23.589540 12214 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
I20260808 21:06:23.590211 12214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.590634 12169 master.cc:537] Master@127.4.177.124:45067 shutting down...
I20260808 21:06:23.591496 12272 mvcc.cc:205] Tried to move back new op lower bound from 7316370155988537344 to 7316370155913629696. Current Snapshot: MvccSnapshot[applied={T|T < 7316370155925917696}]
I20260808 21:06:23.591579 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef890ae9404a288a52f781415b870e. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.591672 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 21:06:23.591712 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.591764 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 21:06:23.591802 12250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } }
I20260808 21:06:23.591845 12250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 21:06:23.592276 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.587729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60290)
0808 21:06:23.587817 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.587827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.587861 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.588272 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.588273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.588275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:23.588317 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.588318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.589321 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.589322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.590518 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.590520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.592002 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.592004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.592074 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.592094 (+ 20us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.592147 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.592205 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 21:06:23.592372 12066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.592468 12066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:23.592579 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.587937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.4.177.125:44519)
0808 21:06:23.587981 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.587983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.587989 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:06:23.588014 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.588015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.588038 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.588094 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:23.588451 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.589225 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.591192 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.591818 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.591890 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.591963 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.592460 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.592502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:23.592839 12123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 LEARNER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260808 21:06:23.593111 11434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e -> Peer 16422c219bfd4e4282697aa1213e3122 (127.4.177.124:45067): Couldn't send request to peer 16422c219bfd4e4282697aa1213e3122. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 21:06:23.593179 12066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 21:06:23.593710 12123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 16422c219bfd4e4282697aa1213e3122 (127.4.177.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.594810 12280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef890ae9404a288a52f781415b870e. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.594920 12280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: LEARNER
I20260808 21:06:23.594815 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: NON_VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: true } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.595566 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: LEARNER
I20260808 21:06:23.596201 12066 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: attempting to promote NON_VOTER 1e3cc4a60dc142ee9018c21bee1b371c to VOTER
I20260808 21:06:23.596418 12066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
I20260808 21:06:23.597343 12123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 LEARNER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 21:06:23.597692 12064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 21:06:23.598135 11434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e -> Peer 16422c219bfd4e4282697aa1213e3122 (127.4.177.124:45067): Couldn't send request to peer 16422c219bfd4e4282697aa1213e3122. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 21:06:23.598533 12064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.599550 12169 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260808 21:06:23.599550 12123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.599674 12169 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 21:06:23.599717 12169 tablet_replica.cc:330] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: stopping tablet replica
I20260808 21:06:23.600553 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.600630 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:06:23.601365 12064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.601456 12064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:23.601557 12064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.601624 12064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:23.613684 12169 master.cc:552] Master@127.4.177.124:45067 shutdown complete.
I20260808 21:06:23.613756 12169 master_runner.cc:305] Clearing existing system tablet
I20260808 21:06:23.613898 12169 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.613961 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.614614 12169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:23.615235 12284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.615428 12169 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.615491 12169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:23.615521 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "16422c219bfd4e4282697aa1213e3122"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f36d4ea769bab015aeda382b98417ff7"
server_key_iv: "fd853c372a003747f30ee4c9c78da769"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.616171 12169 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:23.617568 12169 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 21:06:23.617656 12169 log.cc:1196] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 21:06:23.617986 12169 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Deleting consensus metadata
I20260808 21:06:23.618136 12169 master_runner.cc:315] Copying system tablet from 127.4.177.126:41497
I20260808 21:06:23.618819 12169 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: tablet copy: Beginning tablet copy session from remote peer at address 127.4.177.126:41497
I20260808 21:06:23.632004 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.619179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:34936)
0808 21:06:23.619485 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.619495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.619588 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.619733 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.619734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.619919 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.619966 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.619967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.620649 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.620650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.621194 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.621196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.621570 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.621571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.621626 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.628544 (+ 6918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.628550 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.629062 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.630871 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.630872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.630873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.631056 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.631299 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.631300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.631301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.631363 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.631636 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.631637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.631695 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.631779 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.631903 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":68,"threads_started":1}
I20260808 21:06:23.632000 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.619237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.4.177.126:41497)
0808 21:06:23.619302 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.619305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.619573 (+ 268us) server_negotiation.cc:407] Connection header received
0808 21:06:23.619653 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.619654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.619667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.619727 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.620050 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.620561 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.621282 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.621496 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.621572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.629203 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.629205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.629206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.629213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.630771 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.631133 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.631134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.631135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.631242 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.631445 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.631446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.631446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.631569 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.631645 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.631824 (+ 179us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.631900 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:06:23.632688 11472 tablet_copy_service.cc:140] P 6fef890ae9404a288a52f781415b870e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 16422c219bfd4e4282697aa1213e3122 ({username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:34936)
I20260808 21:06:23.632769 11472 tablet_copy_service.cc:161] P 6fef890ae9404a288a52f781415b870e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 16422c219bfd4e4282697aa1213e3122 at {username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:34936: session id = 16422c219bfd4e4282697aa1213e3122-00000000000000000000000000000000
I20260808 21:06:23.633242 11472 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 21:06:23.633749 12169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 21:06:23.634716 12169 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: tablet copy: Starting download of 0 data blocks...
I20260808 21:06:23.634895 12169 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: tablet copy: Starting download of 1 WAL segments...
I20260808 21:06:23.635751 12169 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 21:06:23.636487 11472 tablet_copy_service.cc:342] P 6fef890ae9404a288a52f781415b870e: Request end of tablet copy session 16422c219bfd4e4282697aa1213e3122-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:34936
I20260808 21:06:23.636543 11472 tablet_copy_service.cc:433] P 6fef890ae9404a288a52f781415b870e: ending tablet copy session 16422c219bfd4e4282697aa1213e3122-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 16422c219bfd4e4282697aa1213e3122
I20260808 21:06:23.638726 12169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.639811 12291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.639986 12292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.640564 12169 server_base.cc:1034] running on GCE node
W20260808 21:06:23.643304 12294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.643469 12169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.643630 12169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.644752 12169 hybrid_clock.cc:648] HybridClock initialized: now 1786223183644746 us; error 24 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.646209 12169 init.cc:377] Logged in from keytab as oryx/127.4.177.124@KRBTEST.COM (short username oryx)
I20260808 21:06:23.646503 12169 webserver.cc:460] Webserver started at http://127.4.177.124:44937/ using document root <none> and password file <none>
I20260808 21:06:23.646627 12169 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.646675 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.647446 12169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:23.648010 12303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.648160 12169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:23.648212 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "16422c219bfd4e4282697aa1213e3122"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f36d4ea769bab015aeda382b98417ff7"
server_key_iv: "fd853c372a003747f30ee4c9c78da769"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.648277 12169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.657392 12169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.659219 12169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.662411 12169 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.124:45067
I20260808 21:06:23.662753 12169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
I20260808 21:06:23.663241 12363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.124:45067 every 8 connection(s)
I20260808 21:06:23.663652 12364 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:06:23.664077 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Bootstrap starting.
I20260808 21:06:23.666006 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:06:23.666245 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Bootstrap complete.
I20260808 21:06:23.666452 12364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
I20260808 21:06:23.666545 12364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16422c219bfd4e4282697aa1213e3122, State: Initialized, Role: LEARNER
I20260808 21:06:23.666592 12364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } }
I20260808 21:06:23.666821 12364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:23.666810 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:23.667022 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: LEARNER
I20260808 21:06:23.668357 12379 catalog_manager.cc:1259] Loaded cluster ID: 5a7f215745cb4ec783165a512c6f5367
I20260808 21:06:23.668452 12379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: loading cluster ID for follower catalog manager: success
I20260808 21:06:23.669534 12379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: acquiring CA information for follower catalog manager: success
I20260808 21:06:23.669831 12379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260808 21:06:23.713928 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:23.714242 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:23.714283 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:23.716017 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:23.716091 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:23.716131 12271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:23.718267 12271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:45637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=45415
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:23.718516 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:23.718778 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.724833 12387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.724807 12389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.724799 12386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.725251 12271 server_base.cc:1034] running on GCE node
I20260808 21:06:23.725472 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.725728 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.726908 12271 hybrid_clock.cc:648] HybridClock initialized: now 1786223183726892 us; error 27 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.730512 12271 init.cc:377] Logged in from keytab as oryx/127.4.177.65@KRBTEST.COM (short username oryx)
I20260808 21:06:23.731114 12271 webserver.cc:460] Webserver started at http://127.4.177.65:45415/ using document root <none> and password file <none>
I20260808 21:06:23.731431 12271 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.731519 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.733552 12271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:23.734671 12396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.734885 12271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:06:23.734985 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "a0a285a4fb32a0835874ef1625f74096"
server_key_iv: "1451968367d2e4b9488c7bd5e5119d46"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.735466 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.746827 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.749287 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:23.749449 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.749686 12271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:23.750007 12271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:23.750041 12271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.750083 12271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:23.750109 12271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.759661 12271 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:45637
I20260808 21:06:23.759704 12524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:45637 every 8 connection(s)
I20260808 21:06:23.760102 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:23.767252 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 12271
I20260808 21:06:23.767427 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11663
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
I20260808 21:06:23.772928 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.760961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:39411 (local address 127.4.177.125:44519)
0808 21:06:23.761021 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.761023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.761359 (+ 336us) server_negotiation.cc:407] Connection header received
0808 21:06:23.762496 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.762497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.762512 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.762566 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.763460 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.764217 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765350 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765639 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.769686 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.769687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.769687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.769696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.771573 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.771949 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.771949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.771950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.772002 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.772209 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.772210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.772210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.772363 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.772405 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.772734 (+ 329us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.772815 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:23.773113 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.761035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:53725 (local address 127.4.177.124:45067)
0808 21:06:23.761474 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.761477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.761485 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:23.762552 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.762553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.762566 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.762644 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.763461 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.764436 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765241 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765538 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765606 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.769244 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.769246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.769246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.769256 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.771278 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.771806 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.771807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.771808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.771872 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.772155 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.772155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.772156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.772302 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.772367 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.772922 (+ 555us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.773010 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":72,"threads_started":1}
I20260808 21:06:23.773748 12529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.760982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.4.177.65:53725)
0808 21:06:23.761263 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.761274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.762402 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.762659 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.762660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.763057 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.763346 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.763352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.764518 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.764519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765143 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.765146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765635 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765781 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.766501 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.766512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.769080 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.771377 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.771386 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.771388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.771710 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.771942 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.771943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.771943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.772081 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.772396 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.772403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.772574 (+ 171us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.772689 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.773490 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":28,"thread_start_us":128,"threads_started":1}
I20260808 21:06:23.773748 12531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.760982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.4.177.65:39411)
0808 21:06:23.761203 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.761249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.762381 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.762639 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.762644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.763091 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.763346 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.763352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.764295 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.764296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765212 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.765214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765735 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765813 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.766596 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.766645 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.769573 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.771660 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.771661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.771661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.771861 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.772076 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.772077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.772077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.772135 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.772450 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.772451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.772574 (+ 123us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.773070 (+ 496us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.773548 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":412,"thread_start_us":113,"threads_started":1}
I20260808 21:06:23.774341 12527 heartbeater.cc:347] Connected to a master server at 127.4.177.125:44519
I20260808 21:06:23.774439 12527 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.774631 12527 heartbeater.cc:511] Master 127.4.177.125:44519 requested a full tablet report, sending...
I20260808 21:06:23.774845 12525 heartbeater.cc:347] Connected to a master server at 127.4.177.124:45067
I20260808 21:06:23.774904 12525 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.774967 12525 heartbeater.cc:511] Master 127.4.177.124:45067 requested a full tablet report, sending...
I20260808 21:06:23.775242 12112 ts_manager.cc:194] Registered new tserver with Master: 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:23.775308 12321 ts_manager.cc:194] Registered new tserver with Master: 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:23.775837 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.761511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:58071 (local address 127.4.177.126:41497)
0808 21:06:23.761567 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.761569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.761579 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:23.762506 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.762507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.762520 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.762554 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.763439 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.763952 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765331 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765532 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765565 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.769670 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.769672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.769672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.769681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.772563 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.773572 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.773573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.773574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.773649 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.774814 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.774815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.774816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.775028 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.775086 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.775576 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.775684 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 21:06:23.776320 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.761313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:58071)
0808 21:06:23.761441 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.761452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.762402 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.762649 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.762650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.763074 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.763345 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.763350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.764065 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.764067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765216 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.765218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.765662 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.765663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.765781 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.766502 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.766512 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.769574 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.772676 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.772677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.772678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.773460 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.773741 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.773742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.773742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.774653 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.775263 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.775264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.775410 (+ 146us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.775992 (+ 582us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.776134 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":6,"thread_start_us":67,"threads_started":1}
I20260808 21:06:23.776715 12526 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:23.776782 12526 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.776865 12526 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:23.777149 11450 ts_manager.cc:194] Re-registered known tserver with Master: 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:23.777554 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:58071
I20260808 21:06:23.784130 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:38891
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:41497
--webserver_port=44455
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:23.869923 12533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:23.870141 12533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:23.870167 12533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:23.871567 12533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:23.871637 12533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:23.871671 12533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:23.873237 12533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:38891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=44455
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:23.873514 12533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:23.873787 12533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:23.878968 12542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.878980 12540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:23.878976 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:23.879367 12533 server_base.cc:1034] running on GCE node
I20260808 21:06:23.879619 12533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:23.879877 12533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:23.881048 12533 hybrid_clock.cc:648] HybridClock initialized: now 1786223183881032 us; error 29 us; skew 500 ppm
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:23.883831 12533 init.cc:377] Logged in from keytab as oryx/127.4.177.66@KRBTEST.COM (short username oryx)
I20260808 21:06:23.884338 12533 webserver.cc:460] Webserver started at http://127.4.177.66:44455/ using document root <none> and password file <none>
I20260808 21:06:23.884600 12533 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:23.884702 12533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:23.886072 12533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 21:06:23.886684 12550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.886864 12533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.886971 12533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef"
format_stamp: "Formatted at 2026-08-08 21:06:22 on dist-test-slave-chsk"
server_key: "82f8f6eeb9db4a834cb071e03c13a99b"
server_key_iv: "dfc421ea0ecd6510c55280571a1c2151"
server_key_version: "encryptionkey@0"
I20260808 21:06:23.887377 12533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:23.908669 12533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:23.911038 12533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:23.911240 12533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:23.911491 12533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:23.911815 12533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:23.911871 12533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.911917 12533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:23.911948 12533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:23.918896 12533 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38891
I20260808 21:06:23.918937 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38891 every 8 connection(s)
I20260808 21:06:23.919374 12533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:23.919876 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 12533
I20260808 21:06:23.920047 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11814
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
Aug 08 21:06:23 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223183, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:23.930824 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:46831
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:41497
--webserver_port=37449
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:06:23.931859 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.921222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:42799 (local address 127.4.177.124:45067)
0808 21:06:23.921351 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.921354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.921516 (+ 162us) server_negotiation.cc:407] Connection header received
0808 21:06:23.921975 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.921977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.921993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.922041 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.923124 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.923923 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.925126 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.925438 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.925507 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.928327 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.928329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.928329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.928338 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.930156 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.930629 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.930631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.930631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.930697 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.931014 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.931015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.931016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.931189 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.931237 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.931638 (+ 401us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.931743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260808 21:06:23.932077 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.920301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.4.177.66:42799)
0808 21:06:23.921430 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.921441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.921854 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.922191 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.922195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.922826 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.923021 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.923025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.924027 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.924028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.925029 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.925031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.925580 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.925582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.925754 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.926320 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.926331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.928229 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.930273 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.930280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.930282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.930508 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.930786 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.930787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.930788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.930932 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.931288 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.931296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.931473 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.931591 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.931785 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260808 21:06:23.932533 12680 heartbeater.cc:347] Connected to a master server at 127.4.177.124:45067
I20260808 21:06:23.932632 12680 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.932864 12680 heartbeater.cc:511] Master 127.4.177.124:45067 requested a full tablet report, sending...
I20260808 21:06:23.933274 12321 ts_manager.cc:194] Registered new tserver with Master: e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66:38891)
I20260808 21:06:23.934278 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.920566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:50683)
0808 21:06:23.925021 (+ 4455us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.925033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.925126 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.925312 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.925314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.925529 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.925618 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.925619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.926479 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.926479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.927281 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.927284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.927908 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.927908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.927971 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.928765 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.928770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.931083 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.933128 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.933130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.933360 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.933573 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933575 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.933575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.933628 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.933931 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.933932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.933975 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.934055 (+ 80us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.934165 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4434,"thread_start_us":457,"threads_started":1}
I20260808 21:06:23.934574 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.920300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.4.177.66:36701)
0808 21:06:23.920703 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:23.920761 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:23.921847 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:23.925551 (+ 3704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:23.925552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:23.925736 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:23.925808 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.925809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.926796 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.926797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.927768 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:23.927771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.928329 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.928330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:23.928407 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.928773 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:23.928777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:23.930926 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:23.933421 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.933422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.933622 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.933873 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:23.933874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:23.933941 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:23.934211 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:23.934212 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:23.934258 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 21:06:23.934337 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:06:23.934447 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":79,"threads_started":1}
I20260808 21:06:23.934659 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.921285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:50683 (local address 127.4.177.126:41497)
0808 21:06:23.921383 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.921385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.925097 (+ 3712us) server_negotiation.cc:407] Connection header received
0808 21:06:23.925191 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.925192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.925212 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.925264 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.925710 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.926376 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.927381 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.927691 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.927736 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.931231 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.931232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.931233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.931241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.933030 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933447 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.933448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.933448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.933503 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933728 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.933729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.933729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.933876 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.933940 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.934480 (+ 540us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.934562 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 21:06:23.935050 12681 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:23.935104 12681 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.935117 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:23.920591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:36701 (local address 127.4.177.125:44519)
0808 21:06:23.925398 (+ 4807us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:23.925400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:23.925409 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:23.925436 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:23.925437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:23.925448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:23.925499 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:23.925901 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.926692 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.927872 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:23.928191 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:23.928232 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:23.931028 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:23.931030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:23.931031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:23.931039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:23.933330 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.933715 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.933717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.933717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.933785 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:23.934004 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:23.934005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:23.934005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:23.934139 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:23.934215 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:23.934942 (+ 727us) server_negotiation.cc:299] Negotiation successful
0808 21:06:23.935033 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4773}
I20260808 21:06:23.935222 12681 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:23.935457 12683 heartbeater.cc:347] Connected to a master server at 127.4.177.125:44519
I20260808 21:06:23.935514 12683 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:23.935513 11450 ts_manager.cc:194] Re-registered known tserver with Master: e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66:38891)
I20260808 21:06:23.935585 12683 heartbeater.cc:511] Master 127.4.177.125:44519 requested a full tablet report, sending...
I20260808 21:06:23.935901 12112 ts_manager.cc:194] Registered new tserver with Master: e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66:38891)
I20260808 21:06:23.935977 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:50683
W20260808 21:06:24.015720 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:24.015950 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:24.015988 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:24.017135 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:24.017201 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:24.017269 12689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:24.018749 12689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:46831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=37449
--tserver_master_addrs=127.4.177.126:41497,127.4.177.125:44519,127.4.177.124:45067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:24.019009 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:24.019276 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:24.024246 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:24.024226 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:24.024219 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:24.024471 12689 server_base.cc:1034] running on GCE node
I20260808 21:06:24.024691 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:24.024969 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:24.026121 12689 hybrid_clock.cc:648] HybridClock initialized: now 1786223184026112 us; error 27 us; skew 500 ppm
Aug 08 21:06:24 dist-test-slave-chsk krb5kdc[11400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223184, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:24.028698 12689 init.cc:377] Logged in from keytab as oryx/127.4.177.67@KRBTEST.COM (short username oryx)
I20260808 21:06:24.029152 12689 webserver.cc:460] Webserver started at http://127.4.177.67:37449/ using document root <none> and password file <none>
I20260808 21:06:24.029376 12689 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:24.029458 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:24.030781 12689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:24.031436 12706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:24.031610 12689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:24.031716 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
format_stamp: "Formatted at 2026-08-08 21:06:23 on dist-test-slave-chsk"
server_key: "f8dcc699dc4e234a29a60a4481be1caa"
server_key_iv: "405fbc780f1d4ce1dd49c1ad55e0829d"
server_key_version: "encryptionkey@0"
I20260808 21:06:24.032032 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:24.048806 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:24.051059 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:24.051285 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:24.051517 12689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:24.051815 12689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:24.051870 12689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:24.051909 12689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:24.051939 12689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:24.058445 12689 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46831
I20260808 21:06:24.058481 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46831 every 8 connection(s)
I20260808 21:06:24.058885 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:06:24 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223184, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
Aug 08 21:06:24 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223184, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.125@KRBTEST.COM
Aug 08 21:06:24 dist-test-slave-chsk krb5kdc[11400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223184, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.124@KRBTEST.COM
I20260808 21:06:24.068621 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 12689
I20260808 21:06:24.069866 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.060009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33497 (local address 127.4.177.126:41497)
0808 21:06:24.060097 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.060099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.060257 (+ 158us) server_negotiation.cc:407] Connection header received
0808 21:06:24.060514 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.060515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.060528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.060562 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.061225 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.061745 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063005 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063340 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063400 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.065814 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.065816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.065816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.065823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.068014 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.068675 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.068677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.068677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.068741 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.069051 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.069052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.069053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.069189 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.069241 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.069659 (+ 418us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.069740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260808 21:06:24.070363 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.4.177.67:42837)
0808 21:06:24.060055 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.060064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.060417 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.061016 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.061017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.061160 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:24.061230 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.061232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.062119 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.062119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.063015 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.063017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063520 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.063582 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.064273 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:24.064278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:24.066799 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:24.068950 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.068952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.068952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.069125 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.069364 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.069365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.069365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.069425 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.069782 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:24.069783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:24.069829 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.069913 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.070109 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":79,"thread_start_us":93,"threads_started":1}
I20260808 21:06:24.070858 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.059649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:42837 (local address 127.4.177.124:45067)
0808 21:06:24.059698 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.059700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.060124 (+ 424us) server_negotiation.cc:407] Connection header received
0808 21:06:24.060515 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.060516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.060529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.060562 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.061301 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.062039 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063075 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063400 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063466 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.066921 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.066923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.066923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.066931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.068870 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.069200 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.069201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.069202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.069258 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.069513 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.069514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.069514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.069674 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.069749 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.070503 (+ 754us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.070736 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:24.071229 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.060030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:46885 (local address 127.4.177.125:44519)
0808 21:06:24.060084 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.060086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.060096 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:24.060504 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.060505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.060520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.060570 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.061204 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.061981 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.062922 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063228 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063298 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.066333 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.066334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.066335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.066343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.068227 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.069851 (+ 1624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.069852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.069852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.069915 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.070623 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.070624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.070624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.070802 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.070845 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.071035 (+ 190us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.071114 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:24.071218 12830 heartbeater.cc:347] Connected to a master server at 127.4.177.124:45067
I20260808 21:06:24.071311 12830 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:24.071512 12830 heartbeater.cc:511] Master 127.4.177.124:45067 requested a full tablet report, sending...
I20260808 21:06:24.071630 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.4.177.67:46885)
0808 21:06:24.059816 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.059849 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.060393 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.060642 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.060646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.060922 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:24.061110 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.061114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.062093 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.062094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.062827 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.062829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063328 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.063478 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.064122 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:24.064126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:24.066245 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:24.068342 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.068343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.068343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.069764 (+ 1421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.070000 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.070001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.070002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.070063 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.070894 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:24.070895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:24.070935 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.071374 (+ 439us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.071504 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260808 21:06:24.071960 12832 heartbeater.cc:347] Connected to a master server at 127.4.177.125:44519
I20260808 21:06:24.072003 12832 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:24.072002 12321 ts_manager.cc:194] Registered new tserver with Master: 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831)
I20260808 21:06:24.072067 12832 heartbeater.cc:511] Master 127.4.177.125:44519 requested a full tablet report, sending...
I20260808 21:06:24.072340 12112 ts_manager.cc:194] Registered new tserver with Master: 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831)
I20260808 21:06:24.072479 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.059932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:33497)
0808 21:06:24.060171 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.060181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.060417 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.060657 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.060657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.060943 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:24.061112 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.061119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.061848 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.061849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.062919 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.062922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.063401 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.063402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.063478 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.064055 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:24.064065 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:24.065723 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:24.068232 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.068241 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.068243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.068589 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.068812 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:24.068813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:24.068813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:24.068961 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:24.069309 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:24.069317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:24.069489 (+ 172us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.072203 (+ 2714us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.072319 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":57,"threads_started":1}
I20260808 21:06:24.072999 12831 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41497
I20260808 21:06:24.073046 12831 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:24.073113 12831 heartbeater.cc:511] Master 127.4.177.126:41497 requested a full tablet report, sending...
I20260808 21:06:24.073344 11450 ts_manager.cc:194] Re-registered known tserver with Master: 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831)
I20260808 21:06:24.073807 11450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:33497
I20260808 21:06:24.076838 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.073245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.0.0.1:54334)
0808 21:06:24.073317 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.073326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.073358 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.073579 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.073580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.073581 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.073620 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.073621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.074466 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.074467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.075658 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.075661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.076559 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.076560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.076623 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.076626 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.076674 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.076769 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 21:06:24.077039 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.073420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.4.177.124:45067)
0808 21:06:24.073451 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.073453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.073459 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:06:24.073476 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.073477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.073498 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.073547 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.073682 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.074380 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.075758 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.076389 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.076447 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.076523 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.076938 (+ 415us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.076980 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 21:06:24.077374 12334 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260808 21:06:24.077545 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fef890ae9404a288a52f781415b870e. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:24.077644 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: LEARNER
I20260808 21:06:24.079563 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.069748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.4.177.126:41497)
0808 21:06:24.070895 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.070897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.070905 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:24.070925 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.070926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.070940 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.071004 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.071997 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.072761 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.073806 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.074113 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.074168 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.075630 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.075631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.075631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.075638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.078261 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.078617 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.078618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.078619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.078685 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.078843 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.078843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.078844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.078978 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.079114 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.079249 (+ 135us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.079330 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":91,"threads_started":1}
I20260808 21:06:24.080372 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.070439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.4.177.125:44519)
0808 21:06:24.071309 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.071311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.072550 (+ 1239us) server_negotiation.cc:407] Connection header received
0808 21:06:24.072643 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.072645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.072655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.072708 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.073720 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.074391 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.075324 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.075558 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.075622 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.077217 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.077218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.077219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.077227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.079012 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.079453 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.079454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.079455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.079518 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.079748 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.079749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.079749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.079882 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.079973 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.080197 (+ 224us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.080282 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":56,"threads_started":1}
I20260808 21:06:24.081310 11450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:24.083280 11450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:24.084074 12282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: NON_VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: true } } }
I20260808 21:06:24.084200 12282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:24.084559 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.070151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.4.177.124:45067)
0808 21:06:24.070669 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.070671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.071946 (+ 1275us) server_negotiation.cc:407] Connection header received
0808 21:06:24.072924 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.072925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.072935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.075763 (+ 2828us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.076372 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.077316 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.079101 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.079418 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.079491 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.081432 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.081433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.081433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.081440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.083084 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.083480 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.083481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.083482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.083538 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.083749 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.083750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.083750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.083898 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.084002 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.084359 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.084440 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":70,"threads_started":1}
I20260808 21:06:24.085150 12282 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: attempting to promote NON_VOTER 16422c219bfd4e4282697aa1213e3122 to VOTER
I20260808 21:06:24.085300 12282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } }
I20260808 21:06:24.085772 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 21:06:24.085930 12334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 LEARNER]: Refusing update from remote peer 6fef890ae9404a288a52f781415b870e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260808 21:06:24.085958 12281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 21:06:24.086143 12281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [LEADER]: Connected to new peer: Peer: permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 21:06:24.086689 12282 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 16422c219bfd4e4282697aa1213e3122 (127.4.177.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.086825 12122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 16422c219bfd4e4282697aa1213e3122 (127.4.177.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.087353 12334 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 16422c219bfd4e4282697aa1213e3122 (127.4.177.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.087782 12282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.087853 12282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:24.087908 12282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.087944 12282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:24.087992 12282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.088027 12282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:24.088173 12155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.088254 12155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:06:24.088402 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fef890ae9404a288a52f781415b870e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:24.088485 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:06:24.094053 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.089870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38891 (local address 127.0.0.1:44012)
0808 21:06:24.090571 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.090580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.090628 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.090884 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.090885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.090886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.090915 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.090916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.091912 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.091914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.093049 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.093051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.093794 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.093795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.093852 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.093856 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.093924 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.093963 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":85,"threads_started":1}
I20260808 21:06:24.094235 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.090127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.4.177.66:38891)
0808 21:06:24.090373 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.090378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.090635 (+ 257us) server_negotiation.cc:407] Connection header received
0808 21:06:24.090688 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.090690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.090730 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.090810 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.090969 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.091820 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.093144 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.093702 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.093794 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.093870 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.093939 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.093974 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260808 21:06:24.094321 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.089775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46831 (local address 127.0.0.1:36318)
0808 21:06:24.090023 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.090033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.090100 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.090926 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.090927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.090928 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.090976 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.090978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.092265 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.092266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.093059 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.093061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.094070 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.094071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.094136 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.094139 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.094172 (+ 33us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.094232 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":17}
I20260808 21:06:24.094429 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.089861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.4.177.67:46831)
0808 21:06:24.090747 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.090761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.090770 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:24.090805 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.090807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.090821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.091049 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.091103 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.092177 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.093151 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.093961 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.094217 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.094291 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.094315 (+ 24us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.094357 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":95,"threads_started":1}
I20260808 21:06:24.094480 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.089811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:45637 (local address 127.0.0.1:36714)
0808 21:06:24.090432 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.090442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.090492 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.090754 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.090756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.090758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.090810 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.090812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.092090 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.092091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.093195 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.093197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.094070 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.094071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.094136 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.094141 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.094172 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.094409 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260808 21:06:24.094801 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.090127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.4.177.65:45637)
0808 21:06:24.090536 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.090540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.090552 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:24.090637 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.090639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.090658 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.090745 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.090879 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.092009 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.093290 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.093964 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.094571 (+ 607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.094655 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.094681 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.094721 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":100,"threads_started":1}
I20260808 21:06:24.095389 12757 tablet_service.cc:1467] Processing CreateTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 (DEFAULT_TABLE table=test-table [id=0f361aa56dbc4cc49f4c8ecb4fc2d9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:24.095428 12605 tablet_service.cc:1467] Processing CreateTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 (DEFAULT_TABLE table=test-table [id=0f361aa56dbc4cc49f4c8ecb4fc2d9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:24.095789 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20de3dec1ff4dda8094b3908a485fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:24.095813 12605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20de3dec1ff4dda8094b3908a485fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:24.095816 12450 tablet_service.cc:1467] Processing CreateTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 (DEFAULT_TABLE table=test-table [id=0f361aa56dbc4cc49f4c8ecb4fc2d9f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:24.096153 12450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20de3dec1ff4dda8094b3908a485fe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:24.098920 12860 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Bootstrap starting.
I20260808 21:06:24.099710 12860 tablet_bootstrap.cc:654] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:24.099884 12861 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Bootstrap starting.
I20260808 21:06:24.100116 12862 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Bootstrap starting.
I20260808 21:06:24.100414 12860 log.cc:824] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:24.100509 12861 tablet_bootstrap.cc:654] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:24.100719 12862 tablet_bootstrap.cc:654] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:24.100842 12861 log.cc:824] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:24.101061 12862 log.cc:824] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:24.101388 12860 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: No bootstrap required, opened a new log
I20260808 21:06:24.101466 12860 ts_tablet_manager.cc:1397] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 21:06:24.101634 12861 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: No bootstrap required, opened a new log
I20260808 21:06:24.101701 12861 ts_tablet_manager.cc:1397] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:24.102164 12862 tablet_bootstrap.cc:492] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: No bootstrap required, opened a new log
I20260808 21:06:24.102241 12862 ts_tablet_manager.cc:1397] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:24.103276 12861 raft_consensus.cc:348] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.103425 12861 raft_consensus.cc:374] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:24.103467 12861 raft_consensus.cc:729] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c004b7666fb407cbf2ba6c6a8ce6032, State: Initialized, Role: FOLLOWER
I20260808 21:06:24.103587 12861 consensus_queue.cc:260] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.103638 12860 raft_consensus.cc:348] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.103811 12860 raft_consensus.cc:374] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:24.103856 12860 raft_consensus.cc:729] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b8ee89fc434b6ba2c599afd74bf6ef, State: Initialized, Role: FOLLOWER
I20260808 21:06:24.103856 12861 ts_tablet_manager.cc:1428] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:24.103989 12860 consensus_queue.cc:260] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.104249 12860 ts_tablet_manager.cc:1428] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:24.104357 12681 heartbeater.cc:503] Master 127.4.177.126:41497 was elected leader, sending a full tablet report...
I20260808 21:06:24.104337 12862 raft_consensus.cc:348] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.104475 12862 raft_consensus.cc:374] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:24.104521 12862 raft_consensus.cc:729] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559a2ef67fd14fdaac0be1ffca873bb5, State: Initialized, Role: FOLLOWER
I20260808 21:06:24.104628 12526 heartbeater.cc:503] Master 127.4.177.126:41497 was elected leader, sending a full tablet report...
I20260808 21:06:24.104642 12862 consensus_queue.cc:260] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.104897 12862 ts_tablet_manager.cc:1428] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 21:06:24.106185 12831 heartbeater.cc:503] Master 127.4.177.126:41497 was elected leader, sending a full tablet report...
W20260808 21:06:24.172041 12684 tablet.cc:2369] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:06:24.261327 12528 tablet.cc:2369] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:06:24.310218 12833 tablet.cc:2369] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:24.328864 12868 raft_consensus.cc:482] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:24.328994 12868 raft_consensus.cc:504] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.329489 12868 leader_election.cc:288] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831), 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:24.332360 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.329852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:51671 (local address 127.4.177.65:45637)
0808 21:06:24.329898 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.329900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.329909 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:24.329940 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.329941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.329950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.330010 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.330177 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.330646 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.331643 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.332002 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.332087 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.332096 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.332248 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.332288 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:24.332378 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.329792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:36883 (local address 127.4.177.67:46831)
0808 21:06:24.329856 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.329858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.329867 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:24.329897 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.329898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.329908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.329963 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:24.330150 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.330640 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.331641 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.332016 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.332101 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.332109 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.332269 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.332307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 21:06:24.332389 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.329704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46831 (local address 127.4.177.66:36883)
0808 21:06:24.329765 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.329776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.329825 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.329969 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.329970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.329971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.330066 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.330067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.330735 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.330736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.331550 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.331552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.332109 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.332110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.332190 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.332192 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.332263 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.332302 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":38}
I20260808 21:06:24.332406 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.329774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:45637 (local address 127.4.177.66:51671)
0808 21:06:24.329829 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:24.329840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:24.329883 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:24.330025 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:24.330026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:24.330027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:24.330101 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.330102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.330732 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.330733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.331519 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:24.331521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.332085 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.332086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:24.332168 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.332171 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:24.332241 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:06:24.332281 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 21:06:24.332772 12778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20de3dec1ff4dda8094b3908a485fe6" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" is_pre_election: true
I20260808 21:06:24.332791 12472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20de3dec1ff4dda8094b3908a485fe6" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" is_pre_election: true
I20260808 21:06:24.332931 12472 raft_consensus.cc:2454] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 0.
I20260808 21:06:24.332943 12778 raft_consensus.cc:2454] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 0.
I20260808 21:06:24.333199 12554 leader_election.cc:302] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 559a2ef67fd14fdaac0be1ffca873bb5, e2b8ee89fc434b6ba2c599afd74bf6ef; no voters:
I20260808 21:06:24.333379 12868 raft_consensus.cc:2781] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:24.333417 12868 raft_consensus.cc:482] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:24.333433 12868 raft_consensus.cc:3037] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:24.334120 12868 raft_consensus.cc:504] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.334271 12868 leader_election.cc:288] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 election: Requested vote from peers 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831), 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:24.334440 12472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20de3dec1ff4dda8094b3908a485fe6" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
I20260808 21:06:24.334520 12472 raft_consensus.cc:3037] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:24.335305 12472 raft_consensus.cc:2454] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 1.
I20260808 21:06:24.335433 12778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20de3dec1ff4dda8094b3908a485fe6" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
I20260808 21:06:24.335494 12555 leader_election.cc:302] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c004b7666fb407cbf2ba6c6a8ce6032, e2b8ee89fc434b6ba2c599afd74bf6ef; no voters:
I20260808 21:06:24.335521 12778 raft_consensus.cc:3037] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:24.335604 12868 raft_consensus.cc:2781] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:24.335814 12868 raft_consensus.cc:686] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 LEADER]: Becoming Leader. State: Replica: e2b8ee89fc434b6ba2c599afd74bf6ef, State: Running, Role: LEADER
I20260808 21:06:24.335911 12868 consensus_queue.cc:237] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } }
I20260808 21:06:24.336764 12778 raft_consensus.cc:2454] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 1.
I20260808 21:06:24.337059 11450 catalog_manager.cc:5495] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66). New cstate: current_term: 1 leader_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } health_report { overall_health: HEALTHY } } }
I20260808 21:06:24.379527 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.377018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.4.177.66:38891)
0808 21:06:24.377112 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.377118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.377126 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:24.377163 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.377164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.377181 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.377262 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:24.377424 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.377901 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.378688 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.378901 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.378978 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.379188 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:24.379287 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:24.379386 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.379411 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.379431 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 21:06:24.381381 12472 raft_consensus.cc:1261] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Refusing update from remote peer e2b8ee89fc434b6ba2c599afd74bf6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:24.381381 12778 raft_consensus.cc:1261] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Refusing update from remote peer e2b8ee89fc434b6ba2c599afd74bf6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:24.381752 12870 consensus_queue.cc:1035] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:24.381892 12868 consensus_queue.cc:1035] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:24.385150 12874 mvcc.cc:205] Tried to move back new op lower bound from 7316370163223359488 to 7316370163040378880. Current Snapshot: MvccSnapshot[applied={T|T < 7316370163223359488}]
I20260808 21:06:24.385178 12876 mvcc.cc:205] Tried to move back new op lower bound from 7316370163223359488 to 7316370163040378880. Current Snapshot: MvccSnapshot[applied={T|T < 7316370163223359488}]
I20260808 21:06:24.388649 11450 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:24.388739 11450 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:24.390437 11450 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fef890ae9404a288a52f781415b870e: Sending DeleteTablet for 3 replicas of tablet b20de3dec1ff4dda8094b3908a485fe6
I20260808 21:06:24.390803 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:24 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36318
I20260808 21:06:24.390796 12605 tablet_service.cc:1514] Processing DeleteTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:24 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:44012
I20260808 21:06:24.391062 12883 tablet_replica.cc:330] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: stopping tablet replica
I20260808 21:06:24.391057 12450 tablet_service.cc:1514] Processing DeleteTablet for tablet b20de3dec1ff4dda8094b3908a485fe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:24 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:36714
I20260808 21:06:24.391188 12883 raft_consensus.cc:2229] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:24.391229 12884 tablet_replica.cc:330] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: stopping tablet replica
I20260808 21:06:24.391345 12883 raft_consensus.cc:2258] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:24.391448 12884 raft_consensus.cc:2229] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:24.391611 12884 raft_consensus.cc:2258] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:24.391822 12883 ts_tablet_manager.cc:1905] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:24.392381 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11414
I20260808 21:06:24.392396 12884 ts_tablet_manager.cc:1905] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:24.395973 12884 ts_tablet_manager.cc:1918] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:24.396118 12884 log.cc:1196] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/b20de3dec1ff4dda8094b3908a485fe6
I20260808 21:06:24.396467 12884 ts_tablet_manager.cc:1939] T b20de3dec1ff4dda8094b3908a485fe6 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting consensus metadata
I20260808 21:06:24.396493 12883 ts_tablet_manager.cc:1918] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:24.396584 12883 log.cc:1196] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/b20de3dec1ff4dda8094b3908a485fe6
I20260808 21:06:24.396862 12883 ts_tablet_manager.cc:1939] T b20de3dec1ff4dda8094b3908a485fe6 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting consensus metadata
I20260808 21:06:24.399492 12885 tablet_replica.cc:330] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: stopping tablet replica
I20260808 21:06:24.399678 12885 raft_consensus.cc:2229] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:24.399852 12885 raft_consensus.cc:2258] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:24.400455 12885 ts_tablet_manager.cc:1905] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 21:06:24.400746 12551 connection.cc:535] server connection from 127.0.0.1:44012 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 21:06:24.401077 12710 connection.cc:535] server connection from 127.0.0.1:36318 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 21:06:24.402204 12885 ts_tablet_manager.cc:1918] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:24.402346 12885 log.cc:1196] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/b20de3dec1ff4dda8094b3908a485fe6
I20260808 21:06:24.402694 12885 ts_tablet_manager.cc:1939] T b20de3dec1ff4dda8094b3908a485fe6 P 1c004b7666fb407cbf2ba6c6a8ce6032: Deleting consensus metadata
W20260808 21:06:24.403113 12398 connection.cc:441] server connection from 127.0.0.1:36714 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36714 (request call id 1) could send its response
I20260808 21:06:24.408757 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.391933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.4.177.124:45067)
0808 21:06:24.391980 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.391983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.399359 (+ 7376us) server_negotiation.cc:407] Connection header received
0808 21:06:24.399416 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.399417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.399434 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.399496 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.400362 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.401132 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.402222 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.402538 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.402584 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.404432 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.404434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.404435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.404443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.406647 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.407049 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.407050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.407051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.407124 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.407359 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.407360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.407360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.407483 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.407568 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.408296 (+ 728us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.408390 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:24.412284 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.403630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.4.177.124:45067)
0808 21:06:24.403670 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.403672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.403973 (+ 301us) server_negotiation.cc:407] Connection header received
0808 21:06:24.404068 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.404069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.404080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.404109 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.405358 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.406094 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.407250 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.407549 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.407607 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.409358 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.409359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.409359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.409366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.410998 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.411412 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.411413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.411413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.411466 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.411708 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.411710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.411710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.411841 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.411916 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.412117 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.412192 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:06:24.412395 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:24.403424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60316 (local address 127.4.177.125:44519)
0808 21:06:24.403472 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:24.403475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:24.403923 (+ 448us) server_negotiation.cc:407] Connection header received
0808 21:06:24.404065 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:24.404066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:24.404081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:24.404113 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:24.404804 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.406714 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.407818 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:24.408113 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:24.408159 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:24.409541 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:24.409543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:24.409543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:24.409550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:24.411173 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.411609 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.411610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.411611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.411664 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:24.411884 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:24.411885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:24.411885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:24.411997 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:24.412095 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:24.412202 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 21:06:24.412282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:06:25.386108 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.385768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:50431)
0808 21:06:25.385913 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.386047 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260808 21:06:25.386322 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.385964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:44241)
0808 21:06:25.386099 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.386170 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
W20260808 21:06:25.386350 12681 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41497 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
W20260808 21:06:25.386572 12526 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41497 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.387269 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.386988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:51005)
0808 21:06:25.387134 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.387223 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
W20260808 21:06:25.387450 12831 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41497 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.890866 12897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6fef890ae9404a288a52f781415b870e)
I20260808 21:06:25.891036 12897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } }
I20260808 21:06:25.891372 12898 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6fef890ae9404a288a52f781415b870e)
I20260808 21:06:25.891695 12897 leader_election.cc:288] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497), 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125:44519)
I20260808 21:06:25.891844 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.891550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35418)
0808 21:06:25.891675 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.891721 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260808 21:06:25.891593 12898 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } }
W20260808 21:06:25.891978 12306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.892350 12898 leader_election.cc:288] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497), 16422c219bfd4e4282697aa1213e3122 (127.4.177.124:45067)
I20260808 21:06:25.892532 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.892251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35422)
0808 21:06:25.892445 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.892500 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":40,"threads_started":1}
I20260808 21:06:25.892612 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.892435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35434)
0808 21:06:25.892545 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.892571 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260808 21:06:25.892726 12306 leader_election.cc:334] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497): Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
W20260808 21:06:25.892786 12095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.893244 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.893111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35444)
0808 21:06:25.893165 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.893207 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 21:06:25.893379 12095 leader_election.cc:334] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497): Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.894956 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.891570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:44519 (local address 127.0.0.1:60318)
0808 21:06:25.891767 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.891777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:25.891855 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:25.892087 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:25.892088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:25.892090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:25.892146 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.892148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.893109 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.893110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.894171 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.894174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.894711 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.894712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.894791 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.894797 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 21:06:25.894854 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 21:06:25.894909 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":46,"threads_started":1}
I20260808 21:06:25.895548 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.891698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.4.177.125:44519)
0808 21:06:25.891799 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:25.891802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:25.891840 (+ 38us) server_negotiation.cc:407] Connection header received
0808 21:06:25.891946 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:25.891947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:25.891961 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:25.892021 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:25.892228 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.893010 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.894265 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.894630 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.894681 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.894691 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:25.895440 (+ 749us) server_negotiation.cc:299] Negotiation successful
0808 21:06:25.895474 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260808 21:06:25.895674 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.892640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.4.177.124:45067)
0808 21:06:25.892840 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:25.892842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:25.892850 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:25.892871 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:25.892871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:25.892880 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:25.893160 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:25.893195 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.893789 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.894871 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.895353 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.895407 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.895414 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:25.895550 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 21:06:25.895601 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260808 21:06:25.895752 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16422c219bfd4e4282697aa1213e3122" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" is_pre_election: true
I20260808 21:06:25.895886 12122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16422c219bfd4e4282697aa1213e3122 in term 1.
I20260808 21:06:25.896029 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.892468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:45067 (local address 127.0.0.1:54356)
0808 21:06:25.892661 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.892672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:25.892724 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:25.893012 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:25.893013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:25.893015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:25.893065 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.893067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.893871 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.893872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.894784 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.894786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.895425 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.895426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.895473 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.895477 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:25.895524 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:06:25.895554 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":43,"threads_started":1}
I20260808 21:06:25.896118 12305 leader_election.cc:302] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16422c219bfd4e4282697aa1213e3122, 1e3cc4a60dc142ee9018c21bee1b371c; no voters: 6fef890ae9404a288a52f781415b870e
I20260808 21:06:25.896225 12897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 21:06:25.896278 12897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6fef890ae9404a288a52f781415b870e)
I20260808 21:06:25.896303 12897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:06:25.896291 12334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "16422c219bfd4e4282697aa1213e3122" is_pre_election: true
I20260808 21:06:25.897043 12897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } }
I20260808 21:06:25.897174 12334 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e3cc4a60dc142ee9018c21bee1b371c in current term 2: Already voted for candidate 16422c219bfd4e4282697aa1213e3122 in this term.
I20260808 21:06:25.897192 12897 leader_election.cc:288] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 election: Requested vote from peers 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497), 1e3cc4a60dc142ee9018c21bee1b371c (127.4.177.125:44519)
I20260808 21:06:25.897351 12095 leader_election.cc:302] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e3cc4a60dc142ee9018c21bee1b371c; no voters: 16422c219bfd4e4282697aa1213e3122, 6fef890ae9404a288a52f781415b870e
I20260808 21:06:25.897388 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.897264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35446)
0808 21:06:25.897345 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.897372 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 21:06:25.897455 12898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:06:25.897539 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16422c219bfd4e4282697aa1213e3122" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e3cc4a60dc142ee9018c21bee1b371c"
I20260808 21:06:25.898169 12898 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:06:25.898540 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.898448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35448)
0808 21:06:25.898493 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.898517 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:06:25.898654 12306 leader_election.cc:334] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497): Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
I20260808 21:06:25.898866 12122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16422c219bfd4e4282697aa1213e3122 in term 2.
I20260808 21:06:25.899019 12305 leader_election.cc:302] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16422c219bfd4e4282697aa1213e3122, 1e3cc4a60dc142ee9018c21bee1b371c; no voters: 6fef890ae9404a288a52f781415b870e
I20260808 21:06:25.899114 12897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:06:25.899263 12897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [term 2 LEADER]: Becoming Leader. State: Replica: 16422c219bfd4e4282697aa1213e3122, State: Running, Role: LEADER
I20260808 21:06:25.899320 12897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } }
I20260808 21:06:25.899680 12908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16422c219bfd4e4282697aa1213e3122. Latest consensus state: current_term: 2 leader_uuid: "16422c219bfd4e4282697aa1213e3122" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:25.899785 12908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:25.899978 12910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:25.900449 12910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:25.900549 12910 catalog_manager.cc:1259] Loaded cluster ID: 5a7f215745cb4ec783165a512c6f5367
I20260808 21:06:25.900583 12910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:25.900735 12910 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:25.900830 12910 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Loaded TSK: 0
I20260808 21:06:25.900965 12910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:25.948205 12321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:25.949772 12321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:25.950733 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.950611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35476)
0808 21:06:25.950669 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.950701 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:06:25.950758 12122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [term 2 FOLLOWER]: Refusing update from remote peer 16422c219bfd4e4282697aa1213e3122: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 21:06:25.951028 12908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 21:06:25.951117 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.951013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.0.0.1:35492)
0808 21:06:25.951065 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.951090 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 21:06:25.951268 12306 consensus_peers.cc:487] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 -> Peer 6fef890ae9404a288a52f781415b870e (127.4.177.126:41497): Couldn't send request to peer 6fef890ae9404a288a52f781415b870e. Status: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 21:06:25.951790 12898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16422c219bfd4e4282697aa1213e3122. Latest consensus state: current_term: 2 leader_uuid: "16422c219bfd4e4282697aa1213e3122" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:25.951895 12898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:06:25.952364 12912 mvcc.cc:205] Tried to move back new op lower bound from 7316370169652219904 to 7316370169444196352. Current Snapshot: MvccSnapshot[applied={T|T < 7316370169652219904}]
I20260808 21:06:25.952346 12897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "16422c219bfd4e4282697aa1213e3122" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:25.952436 12897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:25.952591 12898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "16422c219bfd4e4282697aa1213e3122" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6fef890ae9404a288a52f781415b870e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41497 } } peers { permanent_uuid: "1e3cc4a60dc142ee9018c21bee1b371c" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 44519 } attrs { promote: false } } peers { permanent_uuid: "16422c219bfd4e4282697aa1213e3122" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 45067 } attrs { promote: false } } }
I20260808 21:06:25.952665 12898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e3cc4a60dc142ee9018c21bee1b371c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:06:25.957829 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:45637 (local address 127.0.0.1:36726)
0808 21:06:25.954676 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.954688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:25.954730 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:25.954988 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:25.954989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:25.954991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:25.955062 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.955064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.955852 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.955853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.956848 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.956850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.957585 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.957645 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.957649 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:25.957725 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 21:06:25.957766 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260808 21:06:25.957842 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.4.177.65:45637)
0808 21:06:25.954863 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:25.954865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:25.954874 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:25.954902 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:25.954903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:25.954914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:25.954973 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:25.955132 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.955771 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.956943 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.957506 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957588 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.957601 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:25.957734 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 21:06:25.957774 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":74,"threads_started":1}
I20260808 21:06:25.958138 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38891 (local address 127.0.0.1:44028)
0808 21:06:25.954922 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.954931 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:25.954977 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:25.955298 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:25.955299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:25.955301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:25.955344 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.955345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.956255 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.956256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.957144 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.957146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.957905 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.957967 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.957972 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:25.958024 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 21:06:25.958062 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":67,"threads_started":1}
I20260808 21:06:25.958225 12450 tablet_service.cc:1467] Processing CreateTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 (DEFAULT_TABLE table=test-table [id=17cab9cbd4714c7fa0c53ac72f5fb863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:25.958367 12450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326d2ce766b4444a92c92f04a0d959f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:25.958408 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.4.177.66:38891)
0808 21:06:25.954883 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:25.954885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:25.955174 (+ 289us) server_negotiation.cc:407] Connection header received
0808 21:06:25.955204 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:25.955205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:25.955217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:25.955280 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:25.955423 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.956166 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957232 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.957794 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957861 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.957871 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:25.958274 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 21:06:25.958333 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260808 21:06:25.958289 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46831 (local address 127.0.0.1:36328)
0808 21:06:25.954566 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.954578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:25.954654 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:25.955251 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:25.955252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:25.955254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:25.955292 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.955293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.956249 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.956250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.957327 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:25.957329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.958064 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.958065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:25.958133 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.958137 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:25.958184 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:06:25.958223 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":21}
I20260808 21:06:25.958451 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.954562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.4.177.67:46831)
0808 21:06:25.955090 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:25.955092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:25.955100 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:25.955126 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:25.955127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:25.955138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:25.955353 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:25.955412 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.956159 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.957409 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:25.957977 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:25.958044 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:25.958056 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:25.958345 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 21:06:25.958390 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":77,"threads_started":1}
I20260808 21:06:25.958729 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 (DEFAULT_TABLE table=test-table [id=17cab9cbd4714c7fa0c53ac72f5fb863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:25.958850 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326d2ce766b4444a92c92f04a0d959f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:25.958890 12605 tablet_service.cc:1467] Processing CreateTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 (DEFAULT_TABLE table=test-table [id=17cab9cbd4714c7fa0c53ac72f5fb863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:25.959004 12605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 326d2ce766b4444a92c92f04a0d959f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:25.959825 12920 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032: Bootstrap starting.
I20260808 21:06:25.960260 12920 tablet_bootstrap.cc:654] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:25.960429 12921 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Bootstrap starting.
I20260808 21:06:25.960821 12921 tablet_bootstrap.cc:654] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:25.960932 12920 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032: No bootstrap required, opened a new log
I20260808 21:06:25.960987 12920 ts_tablet_manager.cc:1397] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:25.961122 12920 raft_consensus.cc:348] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.961153 12922 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Bootstrap starting.
I20260808 21:06:25.961203 12920 raft_consensus.cc:374] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:25.961225 12920 raft_consensus.cc:729] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c004b7666fb407cbf2ba6c6a8ce6032, State: Initialized, Role: FOLLOWER
I20260808 21:06:25.961285 12920 consensus_queue.cc:260] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.961410 12921 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: No bootstrap required, opened a new log
I20260808 21:06:25.961454 12920 ts_tablet_manager.cc:1428] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:25.961467 12921 ts_tablet_manager.cc:1397] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:25.961498 12922 tablet_bootstrap.cc:654] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:25.961627 12921 raft_consensus.cc:348] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.961752 12921 raft_consensus.cc:374] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:25.961827 12921 raft_consensus.cc:729] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 559a2ef67fd14fdaac0be1ffca873bb5, State: Initialized, Role: FOLLOWER
I20260808 21:06:25.961951 12921 consensus_queue.cc:260] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.962133 12922 tablet_bootstrap.cc:492] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: No bootstrap required, opened a new log
I20260808 21:06:25.962184 12922 ts_tablet_manager.cc:1397] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:25.962284 12922 raft_consensus.cc:348] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.962345 12922 raft_consensus.cc:374] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:25.962379 12922 raft_consensus.cc:729] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b8ee89fc434b6ba2c599afd74bf6ef, State: Initialized, Role: FOLLOWER
I20260808 21:06:25.962427 12922 consensus_queue.cc:260] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:25.962541 12921 ts_tablet_manager.cc:1428] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:25.962572 12922 ts_tablet_manager.cc:1428] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:25.962739 12830 heartbeater.cc:503] Master 127.4.177.124:45067 was elected leader, sending a full tablet report...
I20260808 21:06:25.962967 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.962352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:37473)
0808 21:06:25.962893 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.962932 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":86,"threads_started":1}
I20260808 21:06:25.963390 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.963222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:50501)
0808 21:06:25.963336 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.963363 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260808 21:06:25.963742 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.961781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:47465)
0808 21:06:25.963622 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.963673 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":89,"threads_started":1}
I20260808 21:06:25.963900 12525 heartbeater.cc:503] Master 127.4.177.124:45067 was elected leader, sending a full tablet report...
I20260808 21:06:25.964095 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.964006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:54317)
0808 21:06:25.964040 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.964073 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260808 21:06:25.964134 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.962729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:35513)
0808 21:06:25.963947 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.964095 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1191,"thread_start_us":77,"threads_started":1}
I20260808 21:06:25.964560 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:25.964470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:57009)
0808 21:06:25.964519 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:25.964543 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 21:06:26.011221 12930 raft_consensus.cc:482] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:26.011340 12930 raft_consensus.cc:504] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:26.011546 12930 leader_election.cc:288] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831), 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:26.011804 12778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326d2ce766b4444a92c92f04a0d959f0" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" is_pre_election: true
I20260808 21:06:26.011804 12472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326d2ce766b4444a92c92f04a0d959f0" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" is_pre_election: true
I20260808 21:06:26.011904 12472 raft_consensus.cc:2454] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 0.
I20260808 21:06:26.011904 12778 raft_consensus.cc:2454] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 0.
I20260808 21:06:26.012107 12555 leader_election.cc:302] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c004b7666fb407cbf2ba6c6a8ce6032, e2b8ee89fc434b6ba2c599afd74bf6ef; no voters:
I20260808 21:06:26.012233 12930 raft_consensus.cc:2781] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:26.012281 12930 raft_consensus.cc:482] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:26.012310 12930 raft_consensus.cc:3037] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:26.013038 12930 raft_consensus.cc:504] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:26.013185 12930 leader_election.cc:288] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 election: Requested vote from peers 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831), 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637)
I20260808 21:06:26.013366 12472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326d2ce766b4444a92c92f04a0d959f0" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032"
I20260808 21:06:26.013423 12778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "326d2ce766b4444a92c92f04a0d959f0" candidate_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "559a2ef67fd14fdaac0be1ffca873bb5"
I20260808 21:06:26.013459 12472 raft_consensus.cc:3037] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:26.013487 12778 raft_consensus.cc:3037] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:26.014070 12472 raft_consensus.cc:2454] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 1.
I20260808 21:06:26.014156 12778 raft_consensus.cc:2454] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b8ee89fc434b6ba2c599afd74bf6ef in term 1.
I20260808 21:06:26.014297 12555 leader_election.cc:302] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c004b7666fb407cbf2ba6c6a8ce6032, e2b8ee89fc434b6ba2c599afd74bf6ef; no voters:
I20260808 21:06:26.014415 12930 raft_consensus.cc:2781] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:26.014596 12930 raft_consensus.cc:686] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 LEADER]: Becoming Leader. State: Replica: e2b8ee89fc434b6ba2c599afd74bf6ef, State: Running, Role: LEADER
I20260808 21:06:26.014686 12680 heartbeater.cc:503] Master 127.4.177.124:45067 was elected leader, sending a full tablet report...
I20260808 21:06:26.014665 12930 consensus_queue.cc:237] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } }
I20260808 21:06:26.015025 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.014892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:51703)
0808 21:06:26.014935 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.014994 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 21:06:26.015543 12321 catalog_manager.cc:5495] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66). New cstate: current_term: 1 leader_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b8ee89fc434b6ba2c599afd74bf6ef" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:26.031244 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.029172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.4.177.66:38891)
0808 21:06:26.029215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:26.029217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:26.029228 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:26.029259 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:26.029259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:26.029268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:26.029313 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:26.029424 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:26.029939 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:26.030618 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:26.030799 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:26.030860 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:26.030972 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:26.030996 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:26.031032 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:26.031081 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 21:06:26.031099 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:06:26.032143 12778 raft_consensus.cc:1261] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Refusing update from remote peer e2b8ee89fc434b6ba2c599afd74bf6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:26.032143 12472 raft_consensus.cc:1261] T 326d2ce766b4444a92c92f04a0d959f0 P 1c004b7666fb407cbf2ba6c6a8ce6032 [term 1 FOLLOWER]: Refusing update from remote peer e2b8ee89fc434b6ba2c599afd74bf6ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:26.032414 12933 consensus_queue.cc:1035] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "559a2ef67fd14fdaac0be1ffca873bb5" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:26.032541 12930 consensus_queue.cc:1035] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c004b7666fb407cbf2ba6c6a8ce6032" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:26.033337 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.033075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:58881)
0808 21:06:26.033251 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.033294 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 21:06:26.033488 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.033318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:50385)
0808 21:06:26.033427 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.033453 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260808 21:06:26.034088 12936 mvcc.cc:205] Tried to move back new op lower bound from 7316370169985888256 to 7316370169917562880. Current Snapshot: MvccSnapshot[applied={T|T < 7316370169985888256}]
I20260808 21:06:26.034269 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.034135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:34953)
0808 21:06:26.034215 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.034243 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 21:06:26.034863 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.034777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.67:40717)
0808 21:06:26.034810 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.034837 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 21:06:26.035097 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.034986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.65:51757)
0808 21:06:26.035042 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.035064 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 21:06:26.035465 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.035331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41497 (local address 127.4.177.66:43289)
0808 21:06:26.035406 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.035433 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 21:06:26.036825 12320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:26.036900 12320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:26.038177 12320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16422c219bfd4e4282697aa1213e3122: Sending DeleteTablet for 3 replicas of tablet 326d2ce766b4444a92c92f04a0d959f0
I20260808 21:06:26.038429 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:26 UTC) from {username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:36328
I20260808 21:06:26.038468 12605 tablet_service.cc:1514] Processing DeleteTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:26 UTC) from {username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:44028
I20260808 21:06:26.038676 12948 tablet_replica.cc:330] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: stopping tablet replica
I20260808 21:06:26.038743 12948 raft_consensus.cc:2229] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:26.038805 12450 tablet_service.cc:1514] Processing DeleteTablet for tablet 326d2ce766b4444a92c92f04a0d959f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:26 UTC) from {username='oryx', principal='oryx/127.4.177.124@KRBTEST.COM'} at 127.0.0.1:36726
I20260808 21:06:26.038823 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 12271
I20260808 21:06:26.043252 12948 raft_consensus.cc:2258] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:26.043401 12947 tablet_replica.cc:330] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: stopping tablet replica
I20260808 21:06:26.043481 12947 raft_consensus.cc:2229] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:26.043540 12947 raft_consensus.cc:2258] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:26.043568 12948 ts_tablet_manager.cc:1905] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:26.043848 12947 ts_tablet_manager.cc:1905] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:26.045581 12947 ts_tablet_manager.cc:1918] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:26.045581 12948 ts_tablet_manager.cc:1918] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:26.045647 12947 log.cc:1196] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/326d2ce766b4444a92c92f04a0d959f0
I20260808 21:06:26.045647 12948 log.cc:1196] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/326d2ce766b4444a92c92f04a0d959f0
I20260808 21:06:26.045907 12947 ts_tablet_manager.cc:1939] T 326d2ce766b4444a92c92f04a0d959f0 P 559a2ef67fd14fdaac0be1ffca873bb5: Deleting consensus metadata
I20260808 21:06:26.045907 12948 ts_tablet_manager.cc:1939] T 326d2ce766b4444a92c92f04a0d959f0 P e2b8ee89fc434b6ba2c599afd74bf6ef: Deleting consensus metadata
I20260808 21:06:26.046223 12306 catalog_manager.cc:4841] TS 559a2ef67fd14fdaac0be1ffca873bb5 (127.4.177.67:46831): tablet 326d2ce766b4444a92c92f04a0d959f0 (table test-table [id=17cab9cbd4714c7fa0c53ac72f5fb863]) successfully deleted
I20260808 21:06:26.046339 12306 catalog_manager.cc:4841] TS e2b8ee89fc434b6ba2c599afd74bf6ef (127.4.177.66:38891): tablet 326d2ce766b4444a92c92f04a0d959f0 (table test-table [id=17cab9cbd4714c7fa0c53ac72f5fb863]) successfully deleted
I20260808 21:06:26.046964 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:26.046853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:45637 (local address 127.0.0.1:36740)
0808 21:06:26.046910 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:26.046938 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:45637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:06:26.047075 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 12533
W20260808 21:06:26.047178 12307 catalog_manager.cc:4584] TS 1c004b7666fb407cbf2ba6c6a8ce6032 (127.4.177.65:45637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 326d2ce766b4444a92c92f04a0d959f0: Network error: Client connection negotiation failed: client connection to 127.4.177.65:45637: connect: Connection refused (error 111)
I20260808 21:06:26.054665 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 12689
I20260808 21:06:26.061484 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 11961
I20260808 21:06:26.067487 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 12169
2026-08-08T21:06:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4819 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:26 dist-test-slave-chsk krb5kdc[12952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:26 dist-test-slave-chsk krb5kdc[12952](info): set up 2 sockets
Aug 08 21:06:26 dist-test-slave-chsk krb5kdc[12952](info): commencing operation
krb5kdc: starting...
W20260808 21:06:27.145169 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab.
Entry for principal oryx/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab.
I20260808 21:06:27.270987 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:32825
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab
--principal=oryx/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:27.361850 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:27.362147 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:27.362193 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:27.363519 12966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:27.363581 12966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 21:06:27.363605 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:27.363628 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:27.363669 12966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:27.363701 12966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:27.365345 12966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:32825
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.126.keytab
--principal=oryx/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:27.365653 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:27.365911 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:27.370894 12975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.370903 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.370894 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:27.371192 12966 server_base.cc:1034] running on GCE node
I20260808 21:06:27.371532 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:27.371851 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:27.373029 12966 hybrid_clock.cc:648] HybridClock initialized: now 1786223187373013 us; error 45 us; skew 500 ppm
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:27.375725 12966 init.cc:377] Logged in from keytab as oryx/127.4.177.126@KRBTEST.COM (short username oryx)
I20260808 21:06:27.376257 12966 webserver.cc:460] Webserver started at http://127.4.177.126:35607/ using document root <none> and password file <none>
I20260808 21:06:27.376533 12966 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:27.376585 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:27.376686 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:27.377760 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b2c56aae90cb438699cd96edc5e67038"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "188ba56b8d6481c8f20bc0a239753e62"
server_key_iv: "2a1f2abd643145c439aa796d3e75dd68"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.378086 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b2c56aae90cb438699cd96edc5e67038"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "188ba56b8d6481c8f20bc0a239753e62"
server_key_iv: "2a1f2abd643145c439aa796d3e75dd68"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.379482 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:27.380209 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.380403 12966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:27.380501 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b2c56aae90cb438699cd96edc5e67038"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "188ba56b8d6481c8f20bc0a239753e62"
server_key_iv: "2a1f2abd643145c439aa796d3e75dd68"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.380560 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:27.389966 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:27.392066 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:27.392189 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:27.397039 12966 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:32825
I20260808 21:06:27.397157 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:32825 every 8 connection(s)
I20260808 21:06:27.397495 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:27.398236 13049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:27.400749 13049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Bootstrap starting.
I20260808 21:06:27.401410 13049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:27.401750 13049 log.cc:824] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:27.402504 13049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: No bootstrap required, opened a new log
I20260808 21:06:27.404727 13049 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } }
I20260808 21:06:27.404875 13049 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:27.404899 13049 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c56aae90cb438699cd96edc5e67038, State: Initialized, Role: FOLLOWER
I20260808 21:06:27.405084 13049 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } }
I20260808 21:06:27.405154 13049 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:27.405198 13049 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:27.405249 13049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:27.405831 13049 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } }
I20260808 21:06:27.405952 13049 leader_election.cc:302] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2c56aae90cb438699cd96edc5e67038; no voters:
I20260808 21:06:27.406272 13049 leader_election.cc:288] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:27.406347 13052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:27.406522 13052 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [term 1 LEADER]: Becoming Leader. State: Replica: b2c56aae90cb438699cd96edc5e67038, State: Running, Role: LEADER
I20260808 21:06:27.406610 13049 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:27.406679 13052 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } }
I20260808 21:06:27.407064 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 12966
I20260808 21:06:27.407219 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:27.407230 13053 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2c56aae90cb438699cd96edc5e67038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } } }
I20260808 21:06:27.407312 13053 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:27.407384 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c56aae90cb438699cd96edc5e67038. Latest consensus state: current_term: 1 leader_uuid: "b2c56aae90cb438699cd96edc5e67038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c56aae90cb438699cd96edc5e67038" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 32825 } } }
I20260808 21:06:27.407457 4805 external_mini_cluster.cc:1428] Setting key 32a18f41a74eabe2d821ea88135f1448
I20260808 21:06:27.407469 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:27.407915 13057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:27.408455 13057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:27.410236 13057 catalog_manager.cc:1347] Generated new cluster ID: 0479188808f44162bc07a845e0e9b20e
I20260808 21:06:27.410291 13057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:27.418020 13057 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:27.418722 13057 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:27.420742 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.408156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.4.177.126:32825)
0808 21:06:27.408609 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.408619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.408656 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:06:27.408783 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.408790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.408849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.408998 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.409962 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.411403 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.412496 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.413109 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.413206 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.416372 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.416377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.416378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.416393 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.418598 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.419029 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.419033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.419033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.419186 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.419401 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.419401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.419402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.419674 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.419830 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.420344 (+ 514us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.420501 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":79,"threads_started":1}
I20260808 21:06:27.426470 13057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Generated new TSK 0
I20260808 21:06:27.426680 13057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
I20260808 21:06:27.458046 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:32825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:27.542732 13076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:27.542977 13076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:27.543016 13076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:27.544253 13076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:27.544324 13076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:27.544359 13076 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:27.544402 13076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:27.545943 13076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:32825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:27.546212 13076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:27.546452 13076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:27.551577 13082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.551584 13086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:27.551600 13076 server_base.cc:1034] running on GCE node
W20260808 21:06:27.551589 13081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:27.551983 13076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:27.552268 13076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:27.553442 13076 hybrid_clock.cc:648] HybridClock initialized: now 1786223187553418 us; error 35 us; skew 500 ppm
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:27.556293 13076 init.cc:377] Logged in from keytab as oryx/127.4.177.65@KRBTEST.COM (short username oryx)
I20260808 21:06:27.556692 13076 webserver.cc:460] Webserver started at http://127.4.177.65:40651/ using document root <none> and password file <none>
I20260808 21:06:27.556874 13076 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:27.556913 13076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:27.557003 13076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:27.558064 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "2ba6d91a99e24834f33941f07d22899e"
server_key_iv: "c013f5277eefa8cea3d4be44fdfb3966"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.558373 13076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "2ba6d91a99e24834f33941f07d22899e"
server_key_iv: "c013f5277eefa8cea3d4be44fdfb3966"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.559913 13076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:06:27.560708 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.560892 13076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:27.560974 13076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "2ba6d91a99e24834f33941f07d22899e"
server_key_iv: "c013f5277eefa8cea3d4be44fdfb3966"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.561061 13076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:27.585988 13076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:27.588418 13076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:27.588542 13076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:27.589023 13076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:27.589071 13076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.589099 13076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:27.589162 13076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.596786 13076 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:33663
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:27.596872 13221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:33663 every 8 connection(s)
I20260808 21:06:27.597191 13076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:27.600466 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.589619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:49191 (local address 127.4.177.126:32825)
0808 21:06:27.589679 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.589681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.589936 (+ 255us) server_negotiation.cc:407] Connection header received
0808 21:06:27.590783 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.590784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.590796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.590869 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.591864 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.592749 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.594446 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.594656 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.594747 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.597398 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.597400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.597400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.597409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.598890 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.599438 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.599439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.599440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.599500 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.599797 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.599798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.599799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.599918 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.599997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.600280 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.600375 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:06:27.600627 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.589575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.65:49191)
0808 21:06:27.589785 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.589836 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.590620 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.590956 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.590961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.591400 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.591721 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.591729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.592860 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.592861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.594361 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.594365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.594750 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.594750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.594902 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.595510 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.595521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.597285 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.599004 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.599012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.599014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.599349 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.599574 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.599575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.599575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.599707 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.599987 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.599995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.600140 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.600243 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.600400 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260808 21:06:27.601243 13222 heartbeater.cc:347] Connected to a master server at 127.4.177.126:32825
I20260808 21:06:27.601349 13222 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:27.601521 13222 heartbeater.cc:511] Master 127.4.177.126:32825 requested a full tablet report, sending...
I20260808 21:06:27.602078 12999 ts_manager.cc:194] Registered new tserver with Master: d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663)
I20260808 21:06:27.602936 12999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:49191
I20260808 21:06:27.604599 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13076
I20260808 21:06:27.604719 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:27.604952 4805 external_mini_cluster.cc:1428] Setting key 018cf330b3c8621ed9136bda5708a3b4
I20260808 21:06:27.608322 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.602389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:38847 (local address 127.4.177.126:32825)
0808 21:06:27.602430 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.602431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.602526 (+ 95us) server_negotiation.cc:407] Connection header received
0808 21:06:27.602560 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.602560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.602569 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.602635 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.602902 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.603471 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.604182 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.604384 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.604471 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.605520 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.605521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.605522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.605536 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.606858 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.607460 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.607461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.607462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.607529 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.607764 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.607765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.607765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.607914 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.607975 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.608161 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.608229 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:27.608330 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.602329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.65:38847)
0808 21:06:27.602449 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.602459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.602525 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.602637 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.602638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.602790 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.602839 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.602840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.603570 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.603571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.604097 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.604098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.604472 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.604473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.604543 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.605002 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.605007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.605419 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.607058 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.607059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.607059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.607365 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.607610 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.607611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.607611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.607678 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.607977 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.607979 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.608045 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.608124 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.608229 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
I20260808 21:06:27.633747 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:32825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:27.717231 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:27.717454 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:27.717479 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:27.718672 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:27.718717 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:27.718739 13233 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:27.718756 13233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:27.720328 13233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:32825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:27.720582 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:27.720829 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:27.725955 13242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.725939 13239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.726050 13240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:27.726222 13233 server_base.cc:1034] running on GCE node
I20260808 21:06:27.726538 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:27.726740 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:27.727921 13233 hybrid_clock.cc:648] HybridClock initialized: now 1786223187727907 us; error 31 us; skew 500 ppm
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:27.730696 13233 init.cc:377] Logged in from keytab as oryx/127.4.177.66@KRBTEST.COM (short username oryx)
I20260808 21:06:27.731323 13233 webserver.cc:460] Webserver started at http://127.4.177.66:33309/ using document root <none> and password file <none>
I20260808 21:06:27.731590 13233 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:27.731638 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:27.731792 13233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:27.732911 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "e4bc5bf09b9744b8b77b905ad3da1432"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "5cfa9455ff83f7d6dce8910974abe905"
server_key_iv: "b942c047604506bd1314a60768b96337"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.733328 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "e4bc5bf09b9744b8b77b905ad3da1432"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "5cfa9455ff83f7d6dce8910974abe905"
server_key_iv: "b942c047604506bd1314a60768b96337"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.735244 13233 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 21:06:27.736292 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.736572 13233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:27.736688 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "e4bc5bf09b9744b8b77b905ad3da1432"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "5cfa9455ff83f7d6dce8910974abe905"
server_key_iv: "b942c047604506bd1314a60768b96337"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.736757 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:27.757128 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:27.759300 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:27.759464 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:27.760044 13233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:27.760089 13233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.760116 13233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:27.760128 13233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:27.767865 13233 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37565
I20260808 21:06:27.767932 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37565 every 8 connection(s)
I20260808 21:06:27.768334 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:27.769747 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13233
I20260808 21:06:27.769882 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:27.770089 4805 external_mini_cluster.cc:1428] Setting key 76d0be7fd5a9ddfcf6c2bb235e81c32f
I20260808 21:06:27.771266 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.760612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:33807 (local address 127.4.177.126:32825)
0808 21:06:27.760700 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.760703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.760840 (+ 137us) server_negotiation.cc:407] Connection header received
0808 21:06:27.761350 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.761352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.761366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.761453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.762415 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.763300 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.764479 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.764843 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.764944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.767903 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.767904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.767905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.767914 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.769526 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.770074 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.770075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.770076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.770144 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.770475 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.770476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.770476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.770602 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.770656 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.771020 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.771128 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 21:06:27.771482 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.760525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.66:33807)
0808 21:06:27.760703 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.760744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.761244 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.761553 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.761558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.761966 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.762290 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.762299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.763409 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.763410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.764366 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.764370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.764962 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.764964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.765112 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.765696 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.765709 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.767800 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.769657 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.769665 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.769667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.769986 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.770242 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.770244 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.770244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.770381 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.770672 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.770678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.770831 (+ 153us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.770975 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.771208 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260808 21:06:27.772023 13376 heartbeater.cc:347] Connected to a master server at 127.4.177.126:32825
I20260808 21:06:27.772125 13376 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:27.772329 13376 heartbeater.cc:511] Master 127.4.177.126:32825 requested a full tablet report, sending...
I20260808 21:06:27.772717 12999 ts_manager.cc:194] Registered new tserver with Master: e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565)
I20260808 21:06:27.773233 12999 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:33807
I20260808 21:06:27.779343 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.773138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:36299 (local address 127.4.177.126:32825)
0808 21:06:27.773187 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.773190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.773345 (+ 155us) server_negotiation.cc:407] Connection header received
0808 21:06:27.773395 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.773396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.773408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.773471 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.773949 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.774610 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.775366 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.775708 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.775785 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.776686 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.776688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.776688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.776695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.778086 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.778434 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.778435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.778436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.778491 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.778717 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.778718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.778718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.778876 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.778961 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.779161 (+ 200us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.779241 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:06:27.779354 13386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.773130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.66:36299)
0808 21:06:27.773259 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.773269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.773337 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.773527 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.773529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.773772 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.773853 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.773855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.774713 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.774714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.775291 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.775293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.775788 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.775789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.775834 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.776291 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.776295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.776580 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.778174 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.778175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.778175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.778357 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.778582 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.778583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.778583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.778651 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.778952 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.778953 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.779002 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.779105 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.779241 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
I20260808 21:06:27.799484 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:32825
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:27.882928 13388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:27.883193 13388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:27.883235 13388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:27.884461 13388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:27.884522 13388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:27.884557 13388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 21:06:27.884608 13388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:27.886123 13388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.177.126:32825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:27.886395 13388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:27.886641 13388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:27.891891 13396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.891917 13394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:27.891886 13393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:27.891908 13388 server_base.cc:1034] running on GCE node
I20260808 21:06:27.892302 13388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:27.892562 13388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:27.893714 13388 hybrid_clock.cc:648] HybridClock initialized: now 1786223187893693 us; error 40 us; skew 500 ppm
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:27.896315 13388 init.cc:377] Logged in from keytab as oryx/127.4.177.67@KRBTEST.COM (short username oryx)
I20260808 21:06:27.896770 13388 webserver.cc:460] Webserver started at http://127.4.177.67:35109/ using document root <none> and password file <none>
I20260808 21:06:27.896957 13388 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:27.896999 13388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:27.897089 13388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:27.898084 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "213acc511d094786a89d16b7ce152d79"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "a661e6613bbf29858f41c0dc312ba0e0"
server_key_iv: "23b88c5f57e46a27c7fc0dc840dc9c70"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.898389 13388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "213acc511d094786a89d16b7ce152d79"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "a661e6613bbf29858f41c0dc312ba0e0"
server_key_iv: "23b88c5f57e46a27c7fc0dc840dc9c70"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.899801 13388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:27.900542 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.900760 13388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:27.900867 13388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "213acc511d094786a89d16b7ce152d79"
format_stamp: "Formatted at 2026-08-08 21:06:27 on dist-test-slave-chsk"
server_key: "a661e6613bbf29858f41c0dc312ba0e0"
server_key_iv: "23b88c5f57e46a27c7fc0dc840dc9c70"
server_key_version: "encryptionkey@0"
I20260808 21:06:27.900949 13388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:27.923727 13388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:27.926242 13388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:27.926362 13388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:27.927026 13388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:27.927088 13388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.927115 13388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:27.927129 13388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:27.934090 13388 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:33875
I20260808 21:06:27.934161 13530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:33875 every 8 connection(s)
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:27.934520 13388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:27.935585 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13388
I20260808 21:06:27.935689 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:27.935920 4805 external_mini_cluster.cc:1428] Setting key 8c4bcc4b119503afa56beaf61b018aca
I20260808 21:06:27.937885 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.927483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:58067 (local address 127.4.177.126:32825)
0808 21:06:27.927560 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.927562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.927795 (+ 233us) server_negotiation.cc:407] Connection header received
0808 21:06:27.928286 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.928287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.928301 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.928385 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.929305 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.930156 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.931752 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.932068 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.932170 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.934833 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.934834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.934835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.934843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.936556 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.937008 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.937008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.937009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.937063 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.937234 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.937234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.937234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.937345 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.937423 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.937743 (+ 320us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.937821 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:27.938023 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.927473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.67:58067)
0808 21:06:27.927653 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.927694 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.928188 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.928472 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.928476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.928876 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.929175 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.929181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.930258 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.930259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.931655 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.931659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.932173 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.932173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.932273 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.932808 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.932819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.934702 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.936634 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.936641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.936644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.936951 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.937108 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.937109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.937109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.937196 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.937455 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.937462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.937614 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.937702 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.937830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":115,"threads_started":1}
I20260808 21:06:27.938617 13531 heartbeater.cc:347] Connected to a master server at 127.4.177.126:32825
I20260808 21:06:27.938719 13531 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:27.938898 13531 heartbeater.cc:511] Master 127.4.177.126:32825 requested a full tablet report, sending...
I20260808 21:06:27.939280 13000 ts_manager.cc:194] Registered new tserver with Master: 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:27.939769 13000 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:58067
I20260808 21:06:27.944931 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.939702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:51673 (local address 127.4.177.126:32825)
0808 21:06:27.939744 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.939745 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.939803 (+ 58us) server_negotiation.cc:407] Connection header received
0808 21:06:27.939899 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.939899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.939909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.939976 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.940726 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.941268 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.941895 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.942075 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.942134 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.942877 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.942878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.942879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.942885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.944066 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.944367 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.944368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.944368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.944414 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.944575 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.944575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.944576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.944666 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.944716 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.944818 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.944872 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:06:27.944940 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.939591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.4.177.67:51673)
0808 21:06:27.939738 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.939749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.939811 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.939983 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.939984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.940607 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:27.940653 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.940655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.941349 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.941349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.941828 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.941830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.942143 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.942144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.942189 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.942540 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:27.942544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:27.942804 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:27.944146 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.944147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.944147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.944308 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.944471 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:27.944471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:27.944472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:27.944523 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:27.944719 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:27.944719 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:27.944754 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.944804 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.944878 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260808 21:06:27.947739 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 21:06:27.970212 13066 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:27.964813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.4.177.126:32825)
0808 21:06:27.964869 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.964872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.964890 (+ 18us) server_negotiation.cc:407] Connection header received
0808 21:06:27.964961 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.964961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.964972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.965013 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.965575 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.966061 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.966763 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.966940 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.966989 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.970110 (+ 3121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47492: BlockingRecv error: recv got EOF from 127.0.0.1:47492 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 08 21:06:27 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:27.977644 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.970958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.4.177.126:32825)
0808 21:06:27.971016 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.971017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.971080 (+ 63us) server_negotiation.cc:407] Connection header received
0808 21:06:27.971123 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.971124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.971133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.971227 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:27.971645 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.972278 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.972909 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.973147 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.973203 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.975325 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:27.975326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:27.975326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:27.975332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:27.976753 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.977075 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.977076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.977076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.977122 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:27.977285 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:27.977285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:27.977285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:27.977374 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:27.977442 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.977524 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.977577 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:06:27.979244 13000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:27.980882 13000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:27.987450 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33663 (local address 127.0.0.1:60120)
0808 21:06:27.983548 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.983593 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.983658 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.983968 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.983971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.983973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:27.984045 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.984046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985409 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.985410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.986265 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.986267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987090 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.987215 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.987232 (+ 17us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.987317 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.987368 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":37,"thread_start_us":51,"threads_started":1}
I20260808 21:06:27.987500 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.4.177.65:33663)
0808 21:06:27.983794 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.983798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.983808 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:27.983849 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.983851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.983865 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.983970 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:27.984143 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985311 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.986365 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987003 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987079 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.987187 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.987348 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.987391 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":83,"threads_started":1}
I20260808 21:06:27.987918 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:33875 (local address 127.0.0.1:57668)
0808 21:06:27.983781 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.983788 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.983856 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.984105 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.984106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.984108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:27.984153 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.984154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985374 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.985375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.986610 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.986612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987679 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.987743 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.987747 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.987803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.987852 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":41,"thread_start_us":91,"threads_started":1}
I20260808 21:06:27.988220 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.4.177.67:33875)
0808 21:06:27.983693 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.983698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.983862 (+ 164us) server_negotiation.cc:407] Connection header received
0808 21:06:27.983933 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.983936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.983953 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.984026 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:27.984231 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985286 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.986714 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987483 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.987642 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.988084 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.988128 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":106,"threads_started":1}
I20260808 21:06:27.988483 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37565 (local address 127.0.0.1:46784)
0808 21:06:27.983685 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:27.983694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:27.983738 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:27.984263 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:27.984264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:27.984266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:27.984311 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.984312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985250 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.985258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.986832 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:27.986833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987914 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:27.987976 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.987981 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:27.988031 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:06:27.988078 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":34,"thread_start_us":88,"threads_started":1}
I20260808 21:06:27.988493 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:27.983599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.4.177.66:37565)
0808 21:06:27.984108 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:27.984113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:27.984123 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:27.984161 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:27.984164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:27.984179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:27.984259 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:27.984387 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.985162 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.986912 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:27.987825 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:27.987913 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:27.988173 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:27.988202 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 21:06:27.988245 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":93,"threads_started":1}
I20260808 21:06:27.988420 13133 tablet_service.cc:1467] Processing CreateTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 (DEFAULT_TABLE table=test-table [id=3e87586898c746ccb0e51e44ca7c14fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:27.988814 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b518ee12c3f341b7a6306fcac2bf2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:27.988799 13459 tablet_service.cc:1467] Processing CreateTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 (DEFAULT_TABLE table=test-table [id=3e87586898c746ccb0e51e44ca7c14fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:27.989130 13459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b518ee12c3f341b7a6306fcac2bf2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:27.989391 13302 tablet_service.cc:1467] Processing CreateTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 (DEFAULT_TABLE table=test-table [id=3e87586898c746ccb0e51e44ca7c14fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:27.989708 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b518ee12c3f341b7a6306fcac2bf2ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:27.992933 13561 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Bootstrap starting.
I20260808 21:06:27.993216 13563 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Bootstrap starting.
I20260808 21:06:27.993640 13561 tablet_bootstrap.cc:654] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:27.993821 13563 tablet_bootstrap.cc:654] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:27.993996 13561 log.cc:824] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:27.994164 13563 log.cc:824] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:27.994374 13560 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Bootstrap starting.
I20260808 21:06:27.994840 13561 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: No bootstrap required, opened a new log
I20260808 21:06:27.994917 13561 ts_tablet_manager.cc:1397] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:27.995008 13563 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: No bootstrap required, opened a new log
I20260808 21:06:27.995079 13563 ts_tablet_manager.cc:1397] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:27.995097 13560 tablet_bootstrap.cc:654] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:27.995457 13560 log.cc:824] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:27.996273 13560 tablet_bootstrap.cc:492] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: No bootstrap required, opened a new log
I20260808 21:06:27.996349 13560 ts_tablet_manager.cc:1397] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:27.996820 13563 raft_consensus.cc:348] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.997018 13563 raft_consensus.cc:374] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:27.997062 13563 raft_consensus.cc:729] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f5c9a62ac94a1cacb866c67ce5ceca, State: Initialized, Role: FOLLOWER
I20260808 21:06:27.997187 13563 consensus_queue.cc:260] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.997129 13561 raft_consensus.cc:348] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.997284 13561 raft_consensus.cc:374] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:27.997331 13561 raft_consensus.cc:729] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Initialized, Role: FOLLOWER
I20260808 21:06:27.997476 13563 ts_tablet_manager.cc:1428] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:27.997459 13561 consensus_queue.cc:260] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.997611 13222 heartbeater.cc:503] Master 127.4.177.126:32825 was elected leader, sending a full tablet report...
I20260808 21:06:27.997714 13561 ts_tablet_manager.cc:1428] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:27.997852 13531 heartbeater.cc:503] Master 127.4.177.126:32825 was elected leader, sending a full tablet report...
I20260808 21:06:27.998534 13560 raft_consensus.cc:348] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.998700 13560 raft_consensus.cc:374] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:27.998749 13560 raft_consensus.cc:729] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Initialized, Role: FOLLOWER
I20260808 21:06:27.998867 13560 consensus_queue.cc:260] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:27.999213 13560 ts_tablet_manager.cc:1428] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:27.999275 13376 heartbeater.cc:503] Master 127.4.177.126:32825 was elected leader, sending a full tablet report...
W20260808 21:06:28.019035 13377 tablet.cc:2369] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:06:28.097995 13223 tablet.cc:2369] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:28.118352 13568 raft_consensus.cc:482] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:28.118492 13568 raft_consensus.cc:504] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:28.118888 13568 leader_election.cc:288] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565)
I20260808 21:06:28.123229 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.119475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:40785 (local address 127.4.177.66:37565)
0808 21:06:28.119535 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.119538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.119602 (+ 64us) server_negotiation.cc:407] Connection header received
0808 21:06:28.119636 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.119638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.119650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.119708 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:28.119850 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.120573 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122201 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.122776 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122852 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.122861 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.123092 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.123135 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 21:06:28.123231 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.119240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:34025 (local address 127.4.177.65:33663)
0808 21:06:28.119307 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.119310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.119385 (+ 75us) server_negotiation.cc:407] Connection header received
0808 21:06:28.119502 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.119508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.119521 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.119609 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:28.119800 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.120573 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122201 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.122765 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122852 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.122861 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.123093 (+ 232us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.123171 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:06:28.123273 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.119366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37565 (local address 127.4.177.67:40785)
0808 21:06:28.119524 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.119534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.119589 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.119718 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.119718 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.119720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:28.119779 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.119780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.120677 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.120678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.122075 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.122078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.122889 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.122996 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.123003 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.123073 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.123123 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":57,"thread_start_us":107,"threads_started":1}
I20260808 21:06:28.123279 13414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.119218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33663 (local address 127.4.177.67:34025)
0808 21:06:28.119296 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.119311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.119406 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.119617 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.119618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.119620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:28.119719 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.119721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.120677 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.120678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.122075 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.122078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.122889 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.122890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.122996 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.123003 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.123073 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.123120 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":106}
I20260808 21:06:28.123811 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b518ee12c3f341b7a6306fcac2bf2ef7" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" is_pre_election: true
I20260808 21:06:28.123932 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b518ee12c3f341b7a6306fcac2bf2ef7" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" is_pre_election: true
I20260808 21:06:28.123996 13172 raft_consensus.cc:2454] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 0.
I20260808 21:06:28.124064 13324 raft_consensus.cc:2454] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 0.
I20260808 21:06:28.124259 13408 leader_election.cc:302] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, d7f5c9a62ac94a1cacb866c67ce5ceca; no voters:
I20260808 21:06:28.124390 13568 raft_consensus.cc:2781] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:28.124487 13568 raft_consensus.cc:482] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:28.124526 13568 raft_consensus.cc:3037] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.125588 13568 raft_consensus.cc:504] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:28.125744 13568 leader_election.cc:288] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 election: Requested vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565)
I20260808 21:06:28.125989 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b518ee12c3f341b7a6306fcac2bf2ef7" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bc5bf09b9744b8b77b905ad3da1432"
I20260808 21:06:28.125975 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b518ee12c3f341b7a6306fcac2bf2ef7" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
I20260808 21:06:28.126067 13324 raft_consensus.cc:3037] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.126068 13172 raft_consensus.cc:3037] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.126972 13172 raft_consensus.cc:2454] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 1.
I20260808 21:06:28.126972 13324 raft_consensus.cc:2454] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 1.
I20260808 21:06:28.127172 13408 leader_election.cc:302] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:28.127264 13568 raft_consensus.cc:2781] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:28.127442 13568 raft_consensus.cc:686] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 1 LEADER]: Becoming Leader. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Running, Role: LEADER
I20260808 21:06:28.127543 13568 consensus_queue.cc:237] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } }
I20260808 21:06:28.128341 12999 catalog_manager.cc:5495] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 213acc511d094786a89d16b7ce152d79 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "213acc511d094786a89d16b7ce152d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:28.168848 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.166280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.4.177.67:33875)
0808 21:06:28.166346 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.166349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.166358 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:28.166411 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.166412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.166429 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.166510 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:28.166675 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.167179 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.168026 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.168225 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.168303 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.168497 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.168604 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.168693 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.168732 (+ 39us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.168754 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:06:28.170737 13172 raft_consensus.cc:1261] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Refusing update from remote peer 213acc511d094786a89d16b7ce152d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.170738 13324 raft_consensus.cc:1261] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Refusing update from remote peer 213acc511d094786a89d16b7ce152d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.171101 13568 consensus_queue.cc:1035] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.171307 13571 consensus_queue.cc:1035] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.173676 13574 mvcc.cc:205] Tried to move back new op lower bound from 7316370178744619008 to 7316370178571108352. Current Snapshot: MvccSnapshot[applied={T|T < 7316370178744619008}]
I20260808 21:06:28.175716 13576 mvcc.cc:205] Tried to move back new op lower bound from 7316370178744619008 to 7316370178571108352. Current Snapshot: MvccSnapshot[applied={T|T < 7316370178744619008 or (T in {7316370178744619008})}]
I20260808 21:06:28.176964 13000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:28.177053 13000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:28.177806 13000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Sending DeleteTablet for 3 replicas of tablet b518ee12c3f341b7a6306fcac2bf2ef7
I20260808 21:06:28.178231 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:60120
I20260808 21:06:28.178256 13459 tablet_service.cc:1514] Processing DeleteTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:57668
I20260808 21:06:28.178459 13583 tablet_replica.cc:330] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: stopping tablet replica
I20260808 21:06:28.178478 13302 tablet_service.cc:1514] Processing DeleteTablet for tablet b518ee12c3f341b7a6306fcac2bf2ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:46784
I20260808 21:06:28.178574 13583 raft_consensus.cc:2229] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:28.178706 13583 raft_consensus.cc:2258] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.178674 13000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:28.178812 13000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:28.179121 13584 tablet_replica.cc:330] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: stopping tablet replica
I20260808 21:06:28.179252 13584 raft_consensus.cc:2229] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:28.179165 13585 tablet_replica.cc:330] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: stopping tablet replica
I20260808 21:06:28.179524 13583 ts_tablet_manager.cc:1905] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.179651 13585 raft_consensus.cc:2229] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:28.179821 13585 raft_consensus.cc:2258] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.180260 13584 raft_consensus.cc:2258] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.180739 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 7439bb9e99264d36aedf906c92d0b2da (DEFAULT_TABLE table=test-table [id=4b805ce0c63c41ce94f2938d728be782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.180867 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7439bb9e99264d36aedf906c92d0b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.180887 13584 ts_tablet_manager.cc:1905] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.181365 13583 ts_tablet_manager.cc:1918] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:28.181437 13583 log.cc:1196] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/b518ee12c3f341b7a6306fcac2bf2ef7
I20260808 21:06:28.181726 13583 ts_tablet_manager.cc:1939] T b518ee12c3f341b7a6306fcac2bf2ef7 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting consensus metadata
I20260808 21:06:28.182098 13563 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Bootstrap starting.
I20260808 21:06:28.182173 13302 tablet_service.cc:1467] Processing CreateTablet for tablet 7439bb9e99264d36aedf906c92d0b2da (DEFAULT_TABLE table=test-table [id=4b805ce0c63c41ce94f2938d728be782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.182297 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7439bb9e99264d36aedf906c92d0b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.182548 13563 tablet_bootstrap.cc:654] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.183243 13584 ts_tablet_manager.cc:1918] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:28.183303 13584 log.cc:1196] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/b518ee12c3f341b7a6306fcac2bf2ef7
I20260808 21:06:28.183624 13584 ts_tablet_manager.cc:1939] T b518ee12c3f341b7a6306fcac2bf2ef7 P 213acc511d094786a89d16b7ce152d79: Deleting consensus metadata
I20260808 21:06:28.183790 13459 tablet_service.cc:1467] Processing CreateTablet for tablet 7439bb9e99264d36aedf906c92d0b2da (DEFAULT_TABLE table=test-table [id=4b805ce0c63c41ce94f2938d728be782]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.183897 13459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7439bb9e99264d36aedf906c92d0b2da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.185057 13561 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Bootstrap starting.
I20260808 21:06:28.185091 13560 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Bootstrap starting.
I20260808 21:06:28.185389 13561 tablet_bootstrap.cc:654] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.185551 13560 tablet_bootstrap.cc:654] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.186054 13561 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: No bootstrap required, opened a new log
I20260808 21:06:28.186093 13561 ts_tablet_manager.cc:1397] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.186168 12984 catalog_manager.cc:4841] TS 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875): tablet b518ee12c3f341b7a6306fcac2bf2ef7 (table test-table [id=3e87586898c746ccb0e51e44ca7c14fa]) successfully deleted
I20260808 21:06:28.186208 13561 raft_consensus.cc:348] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.186254 13561 raft_consensus.cc:374] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.186267 13561 raft_consensus.cc:729] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.186326 13561 consensus_queue.cc:260] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.186437 13561 ts_tablet_manager.cc:1428] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:28.186586 12986 catalog_manager.cc:4841] TS d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663): tablet b518ee12c3f341b7a6306fcac2bf2ef7 (table test-table [id=3e87586898c746ccb0e51e44ca7c14fa]) successfully deleted
I20260808 21:06:28.187216 13585 ts_tablet_manager.cc:1905] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.188793 13585 ts_tablet_manager.cc:1918] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:28.188876 13585 log.cc:1196] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/b518ee12c3f341b7a6306fcac2bf2ef7
I20260808 21:06:28.189138 13585 ts_tablet_manager.cc:1939] T b518ee12c3f341b7a6306fcac2bf2ef7 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting consensus metadata
I20260808 21:06:28.189623 12986 catalog_manager.cc:4841] TS e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565): tablet b518ee12c3f341b7a6306fcac2bf2ef7 (table test-table [id=3e87586898c746ccb0e51e44ca7c14fa]) successfully deleted
I20260808 21:06:28.189657 13560 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: No bootstrap required, opened a new log
I20260808 21:06:28.189700 13560 ts_tablet_manager.cc:1397] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260808 21:06:28.189848 13560 raft_consensus.cc:348] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.189932 13560 raft_consensus.cc:374] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.189954 13560 raft_consensus.cc:729] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.190001 13560 consensus_queue.cc:260] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.190095 13560 ts_tablet_manager.cc:1428] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:28.191572 13563 tablet_bootstrap.cc:492] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: No bootstrap required, opened a new log
I20260808 21:06:28.191666 13563 ts_tablet_manager.cc:1397] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260808 21:06:28.192035 13563 raft_consensus.cc:348] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.192126 13563 raft_consensus.cc:374] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.192152 13563 raft_consensus.cc:729] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f5c9a62ac94a1cacb866c67ce5ceca, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.192209 13563 consensus_queue.cc:260] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.192353 13563 ts_tablet_manager.cc:1428] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260808 21:06:28.436053 13532 tablet.cc:2369] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:28.488897 13569 raft_consensus.cc:482] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:28.488997 13569 raft_consensus.cc:504] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.489436 13569 leader_election.cc:288] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:28.492425 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.489722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:58583 (local address 127.4.177.65:33663)
0808 21:06:28.489771 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.489773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.489852 (+ 79us) server_negotiation.cc:407] Connection header received
0808 21:06:28.489946 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.489947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.489955 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.490016 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:28.490181 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.490661 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.491697 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.492042 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.492127 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.492136 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.492312 (+ 176us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.492354 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:06:28.492456 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.489700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:43965 (local address 127.4.177.67:33875)
0808 21:06:28.489751 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.489754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.489865 (+ 111us) server_negotiation.cc:407] Connection header received
0808 21:06:28.489938 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.489939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.489955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.490029 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:28.490183 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.490646 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.491728 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.492077 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.492183 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.492192 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.492340 (+ 148us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.492384 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:06:28.492460 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.489629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:33875 (local address 127.4.177.66:43965)
0808 21:06:28.489782 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.489792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.489881 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.490034 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.490035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.490038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:28.490113 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.490115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.490733 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.490734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.491628 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.491631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.492188 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.492189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.492258 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.492262 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.492318 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.492356 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":27,"thread_start_us":63,"threads_started":1}
I20260808 21:06:28.492473 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.489628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33663 (local address 127.4.177.66:58583)
0808 21:06:28.489771 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.489782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.489871 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.490020 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.490021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.490023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:28.490109 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.490111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.490744 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.490745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.491595 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.491599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.492127 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.492128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.492209 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.492212 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.492288 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.492335 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260808 21:06:28.492960 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7439bb9e99264d36aedf906c92d0b2da" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" is_pre_election: true
I20260808 21:06:28.492938 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7439bb9e99264d36aedf906c92d0b2da" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79" is_pre_election: true
I20260808 21:06:28.493072 13172 raft_consensus.cc:2454] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:28.493080 13480 raft_consensus.cc:2454] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:28.493263 13253 leader_election.cc:302] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7f5c9a62ac94a1cacb866c67ce5ceca, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:28.493428 13569 raft_consensus.cc:2781] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:28.493479 13569 raft_consensus.cc:482] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:28.493508 13569 raft_consensus.cc:3037] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.494189 13569 raft_consensus.cc:504] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.494351 13569 leader_election.cc:288] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Requested vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:28.494503 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7439bb9e99264d36aedf906c92d0b2da" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79"
I20260808 21:06:28.494508 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7439bb9e99264d36aedf906c92d0b2da" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
I20260808 21:06:28.494596 13480 raft_consensus.cc:3037] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.494596 13172 raft_consensus.cc:3037] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.495286 13172 raft_consensus.cc:2454] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:28.495386 13480 raft_consensus.cc:2454] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:28.495441 13253 leader_election.cc:302] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7f5c9a62ac94a1cacb866c67ce5ceca, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:28.495523 13569 raft_consensus.cc:2781] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:28.495627 13569 raft_consensus.cc:686] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 LEADER]: Becoming Leader. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Running, Role: LEADER
I20260808 21:06:28.495730 13569 consensus_queue.cc:237] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.496510 13000 catalog_manager.cc:5495] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } health_report { overall_health: UNKNOWN } } }
Aug 08 21:06:28 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223187, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.126@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:28.558898 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.551762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:32825 (local address 127.0.0.1:47516)
0808 21:06:28.551911 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.551922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.551997 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.552323 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.552324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.552627 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:28.552674 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.552676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.553334 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.553335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.553869 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.553870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.554198 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.554199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.554254 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.554778 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:28.554783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:28.556271 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:28.557970 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:28.557977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:28.557979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:28.558182 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:28.558370 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:28.558371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:28.558371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:28.558426 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:28.558631 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:28.558637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:28.558680 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.558733 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.558822 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260808 21:06:28.558965 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.551841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.4.177.126:32825)
0808 21:06:28.552071 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.552073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.552082 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:28.552102 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.552103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.552112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.552202 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:28.552751 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.553234 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.553955 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.554133 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.554183 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.556350 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:28.556351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:28.556352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:28.556358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:28.557870 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:28.558250 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:28.558251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:28.558251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:28.558308 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:28.558479 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:28.558479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:28.558480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:28.558578 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:28.558627 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.558765 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.558873 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260808 21:06:28.561125 13000 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47516:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 21:06:28.561342 13000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:28.562826 13459 tablet_service.cc:1467] Processing CreateTablet for tablet 4a677e713e8249989fe94e58726ce7e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dddf98b0c1443abd6966485fdac3e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:28.562826 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 4a677e713e8249989fe94e58726ce7e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dddf98b0c1443abd6966485fdac3e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:28.563001 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a677e713e8249989fe94e58726ce7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.563001 13459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a677e713e8249989fe94e58726ce7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.564533 13563 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: Bootstrap starting.
I20260808 21:06:28.564611 13302 tablet_service.cc:1467] Processing CreateTablet for tablet 4a677e713e8249989fe94e58726ce7e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37dddf98b0c1443abd6966485fdac3e3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 21:06:28.564687 13561 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: Bootstrap starting.
I20260808 21:06:28.564765 13302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a677e713e8249989fe94e58726ce7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.564970 13563 tablet_bootstrap.cc:654] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.565038 13561 tablet_bootstrap.cc:654] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.565639 13563 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: No bootstrap required, opened a new log
I20260808 21:06:28.565698 13563 ts_tablet_manager.cc:1397] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.565726 13561 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: No bootstrap required, opened a new log
I20260808 21:06:28.565780 13561 ts_tablet_manager.cc:1397] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.565853 13563 raft_consensus.cc:348] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.565896 13561 raft_consensus.cc:348] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.565934 13563 raft_consensus.cc:374] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.565956 13561 raft_consensus.cc:374] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.565958 13563 raft_consensus.cc:729] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f5c9a62ac94a1cacb866c67ce5ceca, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.565984 13561 raft_consensus.cc:729] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.566006 13563 consensus_queue.cc:260] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.566067 13561 consensus_queue.cc:260] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.566102 13563 ts_tablet_manager.cc:1428] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:28.566174 13561 ts_tablet_manager.cc:1428] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:28.566241 13567 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.566336 13567 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.566322 13571 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.566397 13571 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.566471 13560 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: Bootstrap starting.
I20260808 21:06:28.566900 13560 tablet_bootstrap.cc:654] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.567649 13560 tablet_bootstrap.cc:492] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: No bootstrap required, opened a new log
I20260808 21:06:28.567714 13560 ts_tablet_manager.cc:1397] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.567862 13560 raft_consensus.cc:348] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.567932 13560 raft_consensus.cc:374] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.567955 13560 raft_consensus.cc:729] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.568008 13560 consensus_queue.cc:260] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.568096 13560 ts_tablet_manager.cc:1428] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:28.568149 13569 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.568277 13569 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.644461 13569 raft_consensus.cc:482] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:28.644562 13569 raft_consensus.cc:504] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.644774 13569 leader_election.cc:288] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:28.644999 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a677e713e8249989fe94e58726ce7e0" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" is_pre_election: true
I20260808 21:06:28.645021 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a677e713e8249989fe94e58726ce7e0" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79" is_pre_election: true
I20260808 21:06:28.645129 13172 raft_consensus.cc:2454] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:28.645129 13480 raft_consensus.cc:2454] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:28.645296 13251 leader_election.cc:302] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:28.645406 13569 raft_consensus.cc:2781] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:28.645450 13569 raft_consensus.cc:482] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:28.645471 13569 raft_consensus.cc:3037] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.646191 13569 raft_consensus.cc:504] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.646324 13569 leader_election.cc:288] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Requested vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:28.646490 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a677e713e8249989fe94e58726ce7e0" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
I20260808 21:06:28.646504 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a677e713e8249989fe94e58726ce7e0" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79"
I20260808 21:06:28.646581 13172 raft_consensus.cc:3037] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.646581 13480 raft_consensus.cc:3037] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:28.647388 13480 raft_consensus.cc:2454] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:28.647388 13172 raft_consensus.cc:2454] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:28.647571 13253 leader_election.cc:302] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7f5c9a62ac94a1cacb866c67ce5ceca, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:28.647701 13569 raft_consensus.cc:2781] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:28.647789 13569 raft_consensus.cc:686] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 LEADER]: Becoming Leader. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Running, Role: LEADER
I20260808 21:06:28.647847 13569 consensus_queue.cc:237] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.648182 13593 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: TxnStatusTablet state changed. Reason: New leader e4bc5bf09b9744b8b77b905ad3da1432. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.648268 13593 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:28.648490 13607 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 21:06:28.648542 13000 catalog_manager.cc:5495] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:28.648633 13607 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 21:06:28.659507 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.656699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37565 (local address 127.0.0.1:46796)
0808 21:06:28.656782 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.656796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.656852 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.657109 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.657110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.657112 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:28.657169 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.657172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.657896 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.657897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.658498 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.658500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.658887 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.658888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.658953 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.658967 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.659320 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.659331 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.659385 (+ 54us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.659436 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 21:06:28.659510 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.656789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.4.177.66:37565)
0808 21:06:28.656942 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.656944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.656951 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:06:28.656990 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.656991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.657006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.657089 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:28.657246 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.657810 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.658596 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.658813 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.658889 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.659097 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.659253 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.659334 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.659390 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.659411 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260808 21:06:28.660663 13172 raft_consensus.cc:1261] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.660704 13480 raft_consensus.cc:1261] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.660914 13569 consensus_queue.cc:1035] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.661042 13593 consensus_queue.cc:1035] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.661669 13571 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: TxnStatusTablet state changed. Reason: New leader e4bc5bf09b9744b8b77b905ad3da1432. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.661757 13571 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.661723 13567 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: TxnStatusTablet state changed. Reason: New leader e4bc5bf09b9744b8b77b905ad3da1432. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.661808 13567 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.662665 13593 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.662755 13593 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:28.662765 13575 mvcc.cc:205] Tried to move back new op lower bound from 7316370180752052224 to 7316370180701917184. Current Snapshot: MvccSnapshot[applied={T|T < 7316370180752052224 or (T in {7316370180752052224})}]
I20260808 21:06:28.663034 13569 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.663044 13571 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.663105 13569 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P e4bc5bf09b9744b8b77b905ad3da1432: This TxnStatusTablet replica's current role is: LEADER
I20260808 21:06:28.663130 13571 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P 213acc511d094786a89d16b7ce152d79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.663631 13567 tablet_replica.cc:439] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } } }
I20260808 21:06:28.663725 13567 tablet_replica.cc:442] T 4a677e713e8249989fe94e58726ce7e0 P d7f5c9a62ac94a1cacb866c67ce5ceca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 21:06:28.667600 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.664969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.4.177.66:37565)
0808 21:06:28.665003 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.665005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.665051 (+ 46us) server_negotiation.cc:407] Connection header received
0808 21:06:28.665145 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.665146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.665156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.665331 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:28.665357 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.666095 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.666817 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.667109 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.667217 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.667370 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.667404 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.667456 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.667505 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.667525 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:06:28.672971 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.670027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:41077 (local address 127.4.177.66:37565)
0808 21:06:28.670089 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.670091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.670193 (+ 102us) server_negotiation.cc:407] Connection header received
0808 21:06:28.670256 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.670257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.670267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.670332 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:28.670466 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.671208 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.672096 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.672387 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.672458 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.672661 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.672692 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.672744 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.672860 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.672880 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:06:28.672982 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.669991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37565 (local address 127.4.177.66:41077)
0808 21:06:28.670127 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:28.670137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:28.670197 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:28.670343 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:28.670343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:28.670344 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 21:06:28.670410 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.670411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.671307 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.671308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.672012 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:28.672015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.672468 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.672469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:28.672534 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.672549 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 21:06:28.672764 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 21:06:28.672766 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:28.672832 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 21:06:28.672868 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260808 21:06:28.675557 13480 raft_consensus.cc:1261] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.675611 13172 raft_consensus.cc:1261] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:28.675752 13593 consensus_queue.cc:1035] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.675872 13569 consensus_queue.cc:1035] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:28.689559 12998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:28.689673 12998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:28.690399 12998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Sending DeleteTablet for 3 replicas of tablet 7439bb9e99264d36aedf906c92d0b2da
I20260808 21:06:28.690678 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7439bb9e99264d36aedf906c92d0b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:60120
I20260808 21:06:28.690762 13459 tablet_service.cc:1514] Processing DeleteTablet for tablet 7439bb9e99264d36aedf906c92d0b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:57668
I20260808 21:06:28.690913 13301 tablet_service.cc:1514] Processing DeleteTablet for tablet 7439bb9e99264d36aedf906c92d0b2da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:28 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:46784
I20260808 21:06:28.691114 13622 tablet_replica.cc:330] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: stopping tablet replica
I20260808 21:06:28.691186 13622 raft_consensus.cc:2229] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:28.691268 13622 raft_consensus.cc:2258] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.691546 13622 ts_tablet_manager.cc:1905] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.693194 13620 tablet_replica.cc:330] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: stopping tablet replica
I20260808 21:06:28.693251 13620 raft_consensus.cc:2229] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:28.693286 13620 raft_consensus.cc:2258] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.693403 13622 ts_tablet_manager.cc:1918] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:28.693450 13622 log.cc:1196] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/7439bb9e99264d36aedf906c92d0b2da
I20260808 21:06:28.693696 13622 ts_tablet_manager.cc:1939] T 7439bb9e99264d36aedf906c92d0b2da P e4bc5bf09b9744b8b77b905ad3da1432: Deleting consensus metadata
I20260808 21:06:28.693976 12986 catalog_manager.cc:4841] TS e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565): tablet 7439bb9e99264d36aedf906c92d0b2da (table test-table [id=4b805ce0c63c41ce94f2938d728be782]) successfully deleted
I20260808 21:06:28.694267 13620 ts_tablet_manager.cc:1905] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.695951 13620 ts_tablet_manager.cc:1918] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:28.696018 13620 log.cc:1196] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/7439bb9e99264d36aedf906c92d0b2da
I20260808 21:06:28.696235 13620 ts_tablet_manager.cc:1939] T 7439bb9e99264d36aedf906c92d0b2da P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting consensus metadata
I20260808 21:06:28.696632 12986 catalog_manager.cc:4841] TS d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663): tablet 7439bb9e99264d36aedf906c92d0b2da (table test-table [id=4b805ce0c63c41ce94f2938d728be782]) successfully deleted
I20260808 21:06:28.699250 13621 tablet_replica.cc:330] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: stopping tablet replica
I20260808 21:06:28.699326 13621 raft_consensus.cc:2229] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:28.699376 13621 raft_consensus.cc:2258] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:28.700242 13621 ts_tablet_manager.cc:1905] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:28.701960 13621 ts_tablet_manager.cc:1918] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:28.702040 13621 log.cc:1196] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/7439bb9e99264d36aedf906c92d0b2da
I20260808 21:06:28.702261 13621 ts_tablet_manager.cc:1939] T 7439bb9e99264d36aedf906c92d0b2da P 213acc511d094786a89d16b7ce152d79: Deleting consensus metadata
I20260808 21:06:28.702601 12984 catalog_manager.cc:4841] TS 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875): tablet 7439bb9e99264d36aedf906c92d0b2da (table test-table [id=4b805ce0c63c41ce94f2938d728be782]) successfully deleted
Aug 08 21:06:28 dist-test-slave-chsk krb5kdc[12952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 21:06:28 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:28 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 21:06:28.726059 13601 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:28.715018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.4.177.126:32825)
0808 21:06:28.715084 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.715087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.715284 (+ 197us) server_negotiation.cc:407] Connection header received
0808 21:06:28.715367 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.715368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.715376 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.715427 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:28.716154 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.716647 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.717720 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.717907 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.717952 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.725982 (+ 8030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47528: BlockingRecv error: recv got EOF from 127.0.0.1:47528 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 21:06:28 dist-test-slave-chsk krb5kdc[12952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM
I20260808 21:06:28.734560 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:28.727107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.4.177.126:32825)
0808 21:06:28.727188 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:28.727190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:28.727231 (+ 41us) server_negotiation.cc:407] Connection header received
0808 21:06:28.727302 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:28.727303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:28.727310 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:28.727352 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:28.727939 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.728487 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.729460 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:28.729644 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:28.729690 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:28.732030 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:28.732032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:28.732032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:28.732039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:28.733413 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:28.733795 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:28.733796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:28.733797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:28.733853 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:28.734051 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:28.734052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:28.734053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:28.734164 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:28.734275 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:28.734376 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 21:06:28.734454 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:06:28.736374 12998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:28.736524 12998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:28.737965 13301 tablet_service.cc:1467] Processing CreateTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 (DEFAULT_TABLE table=test-table [id=c844eb5326a54afb9989cd7a4e753e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.737977 13133 tablet_service.cc:1467] Processing CreateTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 (DEFAULT_TABLE table=test-table [id=c844eb5326a54afb9989cd7a4e753e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.737977 13459 tablet_service.cc:1467] Processing CreateTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 (DEFAULT_TABLE table=test-table [id=c844eb5326a54afb9989cd7a4e753e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:28.738112 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde2505bca7046aa8e78a5ac12a7b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.738116 13459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde2505bca7046aa8e78a5ac12a7b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.738113 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde2505bca7046aa8e78a5ac12a7b4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:28.739558 13563 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Bootstrap starting.
I20260808 21:06:28.739733 13561 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Bootstrap starting.
I20260808 21:06:28.740022 13563 tablet_bootstrap.cc:654] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.740069 13561 tablet_bootstrap.cc:654] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.740473 13560 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Bootstrap starting.
I20260808 21:06:28.740824 13563 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: No bootstrap required, opened a new log
I20260808 21:06:28.740871 13563 ts_tablet_manager.cc:1397] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.740875 13560 tablet_bootstrap.cc:654] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:28.740984 13561 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: No bootstrap required, opened a new log
I20260808 21:06:28.741024 13561 ts_tablet_manager.cc:1397] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:28.741030 13563 raft_consensus.cc:348] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.741097 13563 raft_consensus.cc:374] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.741117 13563 raft_consensus.cc:729] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f5c9a62ac94a1cacb866c67ce5ceca, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.741144 13561 raft_consensus.cc:348] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.741161 13563 consensus_queue.cc:260] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.741200 13561 raft_consensus.cc:374] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.741223 13561 raft_consensus.cc:729] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.741251 13563 ts_tablet_manager.cc:1428] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:28.741262 13561 consensus_queue.cc:260] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.741354 13561 ts_tablet_manager.cc:1428] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:28.742268 13560 tablet_bootstrap.cc:492] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: No bootstrap required, opened a new log
I20260808 21:06:28.742312 13560 ts_tablet_manager.cc:1397] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:06:28.742430 13560 raft_consensus.cc:348] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.742482 13560 raft_consensus.cc:374] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:28.742496 13560 raft_consensus.cc:729] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Initialized, Role: FOLLOWER
I20260808 21:06:28.742553 13560 consensus_queue.cc:260] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:28.742676 13560 ts_tablet_manager.cc:1428] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:29.040719 13571 raft_consensus.cc:482] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:29.040833 13571 raft_consensus.cc:504] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.041007 13571 leader_election.cc:288] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565)
I20260808 21:06:29.041286 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde2505bca7046aa8e78a5ac12a7b4a5" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" is_pre_election: true
I20260808 21:06:29.041410 13172 raft_consensus.cc:2454] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 0.
I20260808 21:06:29.041344 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde2505bca7046aa8e78a5ac12a7b4a5" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" is_pre_election: true
I20260808 21:06:29.041466 13324 raft_consensus.cc:2454] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 0.
I20260808 21:06:29.041615 13408 leader_election.cc:302] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, d7f5c9a62ac94a1cacb866c67ce5ceca; no voters:
I20260808 21:06:29.041769 13571 raft_consensus.cc:2781] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:29.041826 13571 raft_consensus.cc:482] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:29.041843 13571 raft_consensus.cc:3037] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.042621 13571 raft_consensus.cc:504] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.042758 13571 leader_election.cc:288] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 election: Requested vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565)
I20260808 21:06:29.042950 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde2505bca7046aa8e78a5ac12a7b4a5" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
I20260808 21:06:29.042968 13324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde2505bca7046aa8e78a5ac12a7b4a5" candidate_uuid: "213acc511d094786a89d16b7ce152d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bc5bf09b9744b8b77b905ad3da1432"
I20260808 21:06:29.043041 13324 raft_consensus.cc:3037] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.043042 13172 raft_consensus.cc:3037] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.043881 13172 raft_consensus.cc:2454] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 1.
I20260808 21:06:29.043880 13324 raft_consensus.cc:2454] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213acc511d094786a89d16b7ce152d79 in term 1.
I20260808 21:06:29.044066 13408 leader_election.cc:302] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, d7f5c9a62ac94a1cacb866c67ce5ceca; no voters:
I20260808 21:06:29.044167 13571 raft_consensus.cc:2781] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:29.044277 13571 raft_consensus.cc:686] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 1 LEADER]: Becoming Leader. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Running, Role: LEADER
I20260808 21:06:29.044349 13571 consensus_queue.cc:237] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.044893 12998 catalog_manager.cc:5495] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 213acc511d094786a89d16b7ce152d79 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "213acc511d094786a89d16b7ce152d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } health_report { overall_health: HEALTHY } } }
I20260808 21:06:29.097318 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:29.095109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.4.177.67:33875)
0808 21:06:29.095264 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:29.095266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:29.095274 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:29.095296 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:29.095297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:29.095308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:29.095366 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:29.095488 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:29.095979 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:29.096676 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:29.096872 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:29.096921 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:29.097054 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:29.097080 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:29.097126 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:29.097175 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 21:06:29.097192 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260808 21:06:29.098243 13172 raft_consensus.cc:1261] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Refusing update from remote peer 213acc511d094786a89d16b7ce152d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:29.098248 13324 raft_consensus.cc:1261] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Refusing update from remote peer 213acc511d094786a89d16b7ce152d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:29.098510 13571 consensus_queue.cc:1035] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:29.098675 13639 consensus_queue.cc:1035] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:29.101264 12998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:29.101356 12998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:29.101994 12998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Sending DeleteTablet for 3 replicas of tablet dde2505bca7046aa8e78a5ac12a7b4a5
I20260808 21:06:29.102275 13459 tablet_service.cc:1514] Processing DeleteTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:57668
I20260808 21:06:29.102313 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:60120
I20260808 21:06:29.102366 13621 tablet_replica.cc:330] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: stopping tablet replica
I20260808 21:06:29.102420 13621 raft_consensus.cc:2229] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:29.102407 13301 tablet_service.cc:1514] Processing DeleteTablet for tablet dde2505bca7046aa8e78a5ac12a7b4a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:46784
I20260808 21:06:29.102432 13620 tablet_replica.cc:330] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: stopping tablet replica
I20260808 21:06:29.102476 13621 raft_consensus.cc:2258] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.102484 13622 tablet_replica.cc:330] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: stopping tablet replica
I20260808 21:06:29.102494 13620 raft_consensus.cc:2229] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:29.102535 13622 raft_consensus.cc:2229] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:29.102530 13620 raft_consensus.cc:2258] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.102588 13622 raft_consensus.cc:2258] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.102942 13622 ts_tablet_manager.cc:1905] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:29.102995 13621 ts_tablet_manager.cc:1905] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:29.103226 12998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:29.103403 12998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:29.104562 13301 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec887da2ce741679c2ebf2509469e53 (DEFAULT_TABLE table=test-table [id=c986524a7c284b23809fb4eec5a8265c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:29.104570 13459 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec887da2ce741679c2ebf2509469e53 (DEFAULT_TABLE table=test-table [id=c986524a7c284b23809fb4eec5a8265c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:29.104619 13620 ts_tablet_manager.cc:1905] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:29.104619 13133 tablet_service.cc:1467] Processing CreateTablet for tablet 2ec887da2ce741679c2ebf2509469e53 (DEFAULT_TABLE table=test-table [id=c986524a7c284b23809fb4eec5a8265c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:29.104717 13621 ts_tablet_manager.cc:1918] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:29.104765 13621 log.cc:1196] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/dde2505bca7046aa8e78a5ac12a7b4a5
I20260808 21:06:29.104797 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec887da2ce741679c2ebf2509469e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:29.105015 13621 ts_tablet_manager.cc:1939] T dde2505bca7046aa8e78a5ac12a7b4a5 P 213acc511d094786a89d16b7ce152d79: Deleting consensus metadata
I20260808 21:06:29.105166 13459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec887da2ce741679c2ebf2509469e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:29.105875 13563 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: Bootstrap starting.
I20260808 21:06:29.106241 13620 ts_tablet_manager.cc:1918] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:29.106289 13563 tablet_bootstrap.cc:654] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:29.106300 13620 log.cc:1196] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/dde2505bca7046aa8e78a5ac12a7b4a5
I20260808 21:06:29.106388 13622 ts_tablet_manager.cc:1918] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:29.106446 13622 log.cc:1196] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/dde2505bca7046aa8e78a5ac12a7b4a5
I20260808 21:06:29.106645 13620 ts_tablet_manager.cc:1939] T dde2505bca7046aa8e78a5ac12a7b4a5 P d7f5c9a62ac94a1cacb866c67ce5ceca: Deleting consensus metadata
I20260808 21:06:29.106694 13622 ts_tablet_manager.cc:1939] T dde2505bca7046aa8e78a5ac12a7b4a5 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting consensus metadata
I20260808 21:06:29.106865 13561 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Bootstrap starting.
I20260808 21:06:29.106953 12986 catalog_manager.cc:4841] TS d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663): tablet dde2505bca7046aa8e78a5ac12a7b4a5 (table test-table [id=c844eb5326a54afb9989cd7a4e753e12]) successfully deleted
I20260808 21:06:29.107007 12984 catalog_manager.cc:4841] TS 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875): tablet dde2505bca7046aa8e78a5ac12a7b4a5 (table test-table [id=c844eb5326a54afb9989cd7a4e753e12]) successfully deleted
I20260808 21:06:29.107054 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ec887da2ce741679c2ebf2509469e53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:29.107203 13561 tablet_bootstrap.cc:654] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:29.107501 12986 catalog_manager.cc:4841] TS e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565): tablet dde2505bca7046aa8e78a5ac12a7b4a5 (table test-table [id=c844eb5326a54afb9989cd7a4e753e12]) successfully deleted
I20260808 21:06:29.107964 13561 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: No bootstrap required, opened a new log
I20260808 21:06:29.108035 13561 ts_tablet_manager.cc:1397] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260808 21:06:29.108146 13561 raft_consensus.cc:348] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.108222 13561 raft_consensus.cc:374] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:29.108254 13561 raft_consensus.cc:729] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213acc511d094786a89d16b7ce152d79, State: Initialized, Role: FOLLOWER
I20260808 21:06:29.108306 13561 consensus_queue.cc:260] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.108397 13561 ts_tablet_manager.cc:1428] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:29.108441 13560 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Bootstrap starting.
I20260808 21:06:29.108763 13563 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: No bootstrap required, opened a new log
I20260808 21:06:29.108785 13560 tablet_bootstrap.cc:654] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:29.108798 13563 ts_tablet_manager.cc:1397] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260808 21:06:29.108907 13563 raft_consensus.cc:348] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.108982 13563 raft_consensus.cc:374] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:29.109004 13563 raft_consensus.cc:729] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f5c9a62ac94a1cacb866c67ce5ceca, State: Initialized, Role: FOLLOWER
I20260808 21:06:29.109047 13563 consensus_queue.cc:260] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.109148 13563 ts_tablet_manager.cc:1428] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:29.109570 13560 tablet_bootstrap.cc:492] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: No bootstrap required, opened a new log
I20260808 21:06:29.109622 13560 ts_tablet_manager.cc:1397] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:29.109740 13560 raft_consensus.cc:348] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.109833 13560 raft_consensus.cc:374] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:29.109865 13560 raft_consensus.cc:729] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Initialized, Role: FOLLOWER
I20260808 21:06:29.109917 13560 consensus_queue.cc:260] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.110009 13560 ts_tablet_manager.cc:1428] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:29.209378 13613 raft_consensus.cc:482] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:29.209515 13613 raft_consensus.cc:504] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.209744 13613 leader_election.cc:288] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:29.209983 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec887da2ce741679c2ebf2509469e53" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" is_pre_election: true
I20260808 21:06:29.209983 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec887da2ce741679c2ebf2509469e53" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79" is_pre_election: true
I20260808 21:06:29.210098 13172 raft_consensus.cc:2454] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:29.210098 13480 raft_consensus.cc:2454] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 0.
I20260808 21:06:29.210270 13251 leader_election.cc:302] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:29.210394 13613 raft_consensus.cc:2781] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:29.210453 13613 raft_consensus.cc:482] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:29.210485 13613 raft_consensus.cc:3037] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.211308 13613 raft_consensus.cc:504] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.211453 13613 leader_election.cc:288] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Requested vote from peers d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663), 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875)
I20260808 21:06:29.211634 13480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec887da2ce741679c2ebf2509469e53" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213acc511d094786a89d16b7ce152d79"
I20260808 21:06:29.211707 13480 raft_consensus.cc:3037] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.211656 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ec887da2ce741679c2ebf2509469e53" candidate_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca"
I20260808 21:06:29.211745 13172 raft_consensus.cc:3037] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:29.212322 13480 raft_consensus.cc:2454] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:29.212404 13172 raft_consensus.cc:2454] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bc5bf09b9744b8b77b905ad3da1432 in term 1.
I20260808 21:06:29.212502 13251 leader_election.cc:302] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 213acc511d094786a89d16b7ce152d79, e4bc5bf09b9744b8b77b905ad3da1432; no voters:
I20260808 21:06:29.212617 13613 raft_consensus.cc:2781] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:29.212765 13613 raft_consensus.cc:686] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 LEADER]: Becoming Leader. State: Replica: e4bc5bf09b9744b8b77b905ad3da1432, State: Running, Role: LEADER
I20260808 21:06:29.212828 13613 consensus_queue.cc:237] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } }
I20260808 21:06:29.213459 12998 catalog_manager.cc:5495] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bc5bf09b9744b8b77b905ad3da1432" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 } health_report { overall_health: UNKNOWN } } }
I20260808 21:06:29.228031 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:29.225901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.4.177.66:37565)
0808 21:06:29.226000 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:29.226002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:29.226011 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:29.226033 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:29.226034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:29.226045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:29.226125 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:29.226241 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:29.226720 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:29.227462 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:29.227644 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:29.227699 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:29.227802 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:29.227824 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:29.227857 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:29.227903 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 21:06:29.227921 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
I20260808 21:06:29.230110 13172 raft_consensus.cc:1261] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:29.230219 13480 raft_consensus.cc:1261] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Refusing update from remote peer e4bc5bf09b9744b8b77b905ad3da1432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:29.230422 13652 consensus_queue.cc:1035] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f5c9a62ac94a1cacb866c67ce5ceca" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:29.230512 13649 consensus_queue.cc:1035] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213acc511d094786a89d16b7ce152d79" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:29.240644 12998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:29.240736 12998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:29.241338 12998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c56aae90cb438699cd96edc5e67038: Sending DeleteTablet for 3 replicas of tablet 2ec887da2ce741679c2ebf2509469e53
I20260808 21:06:29.241611 13459 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec887da2ce741679c2ebf2509469e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:57668
I20260808 21:06:29.241612 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec887da2ce741679c2ebf2509469e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:60120
I20260808 21:06:29.241701 13621 tablet_replica.cc:330] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: stopping tablet replica
I20260808 21:06:29.241711 13620 tablet_replica.cc:330] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca: stopping tablet replica
I20260808 21:06:29.241753 13621 raft_consensus.cc:2229] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:29.241763 13620 raft_consensus.cc:2229] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:29.241784 13621 raft_consensus.cc:2258] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.241801 13620 raft_consensus.cc:2258] T 2ec887da2ce741679c2ebf2509469e53 P d7f5c9a62ac94a1cacb866c67ce5ceca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.241822 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13076
I20260808 21:06:29.243682 13301 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ec887da2ce741679c2ebf2509469e53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:29 UTC) from {username='oryx', principal='oryx/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:46784
I20260808 21:06:29.243759 13621 ts_tablet_manager.cc:1905] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:29.243783 13622 tablet_replica.cc:330] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: stopping tablet replica
I20260808 21:06:29.243831 13622 raft_consensus.cc:2229] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:29.243891 13622 raft_consensus.cc:2258] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:29.244319 13622 ts_tablet_manager.cc:1905] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:29.245273 13621 ts_tablet_manager.cc:1918] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:29.245348 13621 log.cc:1196] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/2ec887da2ce741679c2ebf2509469e53
I20260808 21:06:29.245565 13621 ts_tablet_manager.cc:1939] T 2ec887da2ce741679c2ebf2509469e53 P 213acc511d094786a89d16b7ce152d79: Deleting consensus metadata
I20260808 21:06:29.245886 12984 catalog_manager.cc:4841] TS 213acc511d094786a89d16b7ce152d79 (127.4.177.67:33875): tablet 2ec887da2ce741679c2ebf2509469e53 (table test-table [id=c986524a7c284b23809fb4eec5a8265c]) successfully deleted
I20260808 21:06:29.246132 13622 ts_tablet_manager.cc:1918] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 21:06:29.246210 13622 log.cc:1196] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/2ec887da2ce741679c2ebf2509469e53
I20260808 21:06:29.246424 13622 ts_tablet_manager.cc:1939] T 2ec887da2ce741679c2ebf2509469e53 P e4bc5bf09b9744b8b77b905ad3da1432: Deleting consensus metadata
I20260808 21:06:29.246731 12986 catalog_manager.cc:4841] TS e4bc5bf09b9744b8b77b905ad3da1432 (127.4.177.66:37565): tablet 2ec887da2ce741679c2ebf2509469e53 (table test-table [id=c986524a7c284b23809fb4eec5a8265c]) successfully deleted
W20260808 21:06:29.249863 12986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:33663 (error 108)
I20260808 21:06:29.250101 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13233
I20260808 21:06:29.251056 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:29.250208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33663 (local address 127.0.0.1:60128)
0808 21:06:29.250939 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:29.251002 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":37,"threads_started":1}
W20260808 21:06:29.252497 12986 catalog_manager.cc:4584] TS d7f5c9a62ac94a1cacb866c67ce5ceca (127.4.177.65:33663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ec887da2ce741679c2ebf2509469e53: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33663: connect: Connection refused (error 111)
I20260808 21:06:29.259117 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13388
I20260808 21:06:29.266423 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 12966
2026-08-08T21:06:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3197 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:29 dist-test-slave-chsk krb5kdc[13664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:29 dist-test-slave-chsk krb5kdc[13664](info): set up 2 sockets
Aug 08 21:06:29 dist-test-slave-chsk krb5kdc[13664](info): commencing operation
krb5kdc: starting...
W20260808 21:06:30.339359 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:30 dist-test-slave-chsk krb5kdc[13664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:30.458750 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35269
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:30.540978 13677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:30.541245 13677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:30.541289 13677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:30.542522 13677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:30.542577 13677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:30.542599 13677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:30.542623 13677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:30.542672 13677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:30.544250 13677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:30.544528 13677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:30.544752 13677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:30.549496 13687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:30.549510 13685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:30.549508 13684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:30.549853 13677 server_base.cc:1034] running on GCE node
I20260808 21:06:30.550139 13677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:30.550442 13677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:30.551591 13677 hybrid_clock.cc:648] HybridClock initialized: now 1786223190551580 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T21:06:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1287 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:30 dist-test-slave-chsk krb5kdc[13693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:30 dist-test-slave-chsk krb5kdc[13693](info): set up 2 sockets
Aug 08 21:06:30 dist-test-slave-chsk krb5kdc[13693](info): commencing operation
krb5kdc: starting...
W20260808 21:06:31.623323 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:31 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:31.744046 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33233
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:33233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:31.825593 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:31.825793 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:31.825814 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:31.826943 13708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:31.826985 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:31.827000 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:31.827054 13708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:31.827082 13708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:31.828634 13708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:33233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:31.828866 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:31.829092 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:31.833793 13714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:31.833817 13716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:31.833830 13713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:31.834147 13708 server_base.cc:1034] running on GCE node
I20260808 21:06:31.834383 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:31.834688 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:31.835846 13708 hybrid_clock.cc:648] HybridClock initialized: now 1786223191835840 us; error 36 us; skew 500 ppm
Aug 08 21:06:31 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223191, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:31.838223 13708 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:31.838676 13708 webserver.cc:460] Webserver started at http://127.4.177.126:44829/ using document root <none> and password file <none>
I20260808 21:06:31.838907 13708 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:31.838972 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:31.839133 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:31.840215 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "e081182435c54154974819e9f61c6da2"
format_stamp: "Formatted at 2026-08-08 21:06:31 on dist-test-slave-chsk"
server_key: "87f6021c0023ac23255a4d82ec3d1077"
server_key_iv: "24b32f755cced3bedd65a7606c66caf0"
server_key_version: "encryptionkey@0"
I20260808 21:06:31.840564 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "e081182435c54154974819e9f61c6da2"
format_stamp: "Formatted at 2026-08-08 21:06:31 on dist-test-slave-chsk"
server_key: "87f6021c0023ac23255a4d82ec3d1077"
server_key_iv: "24b32f755cced3bedd65a7606c66caf0"
server_key_version: "encryptionkey@0"
I20260808 21:06:31.841841 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:31.842545 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:31.842713 13708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:31.842808 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "e081182435c54154974819e9f61c6da2"
format_stamp: "Formatted at 2026-08-08 21:06:31 on dist-test-slave-chsk"
server_key: "87f6021c0023ac23255a4d82ec3d1077"
server_key_iv: "24b32f755cced3bedd65a7606c66caf0"
server_key_version: "encryptionkey@0"
I20260808 21:06:31.842890 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:31.868767 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:31.870589 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:31.870756 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:31.874759 13708 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:33233
I20260808 21:06:31.874799 13779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:33233 every 8 connection(s)
I20260808 21:06:31.875193 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:31.876046 13780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:31.878345 13780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Bootstrap starting.
I20260808 21:06:31.878926 13780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:31.879073 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13708
I20260808 21:06:31.879218 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:31.879320 13780 log.cc:824] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:31.879426 4805 external_mini_cluster.cc:1428] Setting key addc28362a0986090f7067a8c6173a5d
I20260808 21:06:31.880071 13780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: No bootstrap required, opened a new log
I20260808 21:06:31.882205 13780 raft_consensus.cc:348] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:31.882357 13780 raft_consensus.cc:374] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:31.882408 13780 raft_consensus.cc:729] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e081182435c54154974819e9f61c6da2, State: Initialized, Role: FOLLOWER
I20260808 21:06:31.882532 13780 consensus_queue.cc:260] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:31.882609 13780 raft_consensus.cc:388] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:31.882632 13780 raft_consensus.cc:482] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:31.882696 13780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:31.883338 13780 raft_consensus.cc:504] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:31.883453 13780 leader_election.cc:302] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e081182435c54154974819e9f61c6da2; no voters:
I20260808 21:06:31.883647 13780 leader_election.cc:288] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:31.883683 13786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:31.883805 13786 raft_consensus.cc:686] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 LEADER]: Becoming Leader. State: Replica: e081182435c54154974819e9f61c6da2, State: Running, Role: LEADER
I20260808 21:06:31.883888 13786 consensus_queue.cc:237] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:31.883915 13780 sys_catalog.cc:564] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:31.885447 13788 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e081182435c54154974819e9f61c6da2. Latest consensus state: current_term: 1 leader_uuid: "e081182435c54154974819e9f61c6da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } } }
Aug 08 21:06:31 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:31.885550 13787 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e081182435c54154974819e9f61c6da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } } }
I20260808 21:06:31.886384 13787 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:31.885632 13788 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: This master's current role is: LEADER
W20260808 21:06:31.885908 13801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:06:31.886639 13801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:06:31.886713 13803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:31.886883 13803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:31.888469 13803 catalog_manager.cc:1347] Generated new cluster ID: 3db05f250937484f93cb358e14adf7db
I20260808 21:06:31.888525 13803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:31.890340 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:31.879877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.4.177.126:33233)
0808 21:06:31.880042 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:31.880052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:31.880088 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:06:31.880576 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:31.880582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:31.880631 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:31.880767 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:31.881405 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:31.882274 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:31.883167 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:31.883673 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:31.883805 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:31.886795 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:31.886799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:31.886799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:31.886811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:31.888846 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:31.889195 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:31.889198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:31.889198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:31.889286 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:31.889483 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:31.889484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:31.889484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:31.889720 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:31.889888 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:31.889991 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 21:06:31.890127 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260808 21:06:31.932231 13803 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:31.932806 13803 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:31.939253 13803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Generated new TSK 0
I20260808 21:06:31.939455 13803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:31.966174 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:32.049108 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.049324 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.049347 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.050479 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.050522 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.050568 13808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:32.052009 13808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.052237 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.052461 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.057296 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.057303 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.057302 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.057531 13808 server_base.cc:1034] running on GCE node
I20260808 21:06:32.057734 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.057931 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.059109 13808 hybrid_clock.cc:648] HybridClock initialized: now 1786223192059088 us; error 32 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.061581 13808 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:32.062002 13808 webserver.cc:460] Webserver started at http://127.4.177.65:43099/ using document root <none> and password file <none>
I20260808 21:06:32.062239 13808 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.062304 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.062448 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:32.063555 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "18c8ca724b17448fa030107f0a32198f"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0477f1b7e882b20e78f451b9c7716802"
server_key_iv: "c0431af1f9d7e287c05502bf032e491c"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.063913 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "18c8ca724b17448fa030107f0a32198f"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0477f1b7e882b20e78f451b9c7716802"
server_key_iv: "c0431af1f9d7e287c05502bf032e491c"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.065217 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:32.065958 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.066129 13808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:32.066229 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "18c8ca724b17448fa030107f0a32198f"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0477f1b7e882b20e78f451b9c7716802"
server_key_iv: "c0431af1f9d7e287c05502bf032e491c"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.066298 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.100106 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.103205 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.103386 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.103693 13808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:32.104125 13808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:32.104182 13808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.104233 13808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:32.104265 13808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.112538 13808 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:46425
I20260808 21:06:32.112586 13947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:46425 every 8 connection(s)
I20260808 21:06:32.113075 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:32.121929 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13808
I20260808 21:06:32.122062 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:32.122272 4805 external_mini_cluster.cc:1428] Setting key 2e5ddb9dc2a8982452de7b93ed5b4228
I20260808 21:06:32.122681 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.113945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:45511 (local address 127.4.177.126:33233)
0808 21:06:32.114002 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.114004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.114226 (+ 222us) server_negotiation.cc:407] Connection header received
0808 21:06:32.114746 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.114747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.114760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.114823 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.115828 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.116349 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.117316 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.117579 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.117657 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.119839 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:32.119840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:32.119840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:32.119848 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:32.121253 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.121664 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.121665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.121665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.121725 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.121973 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.121974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.121974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.122138 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:32.122203 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:32.122486 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 21:06:32.122589 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:06:32.122838 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.113939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:45511)
0808 21:06:32.114108 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.114145 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.114646 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.114893 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.114897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.115539 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.115727 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.115732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.116440 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.116442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.117232 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.117234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.117665 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.117666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.117780 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.118296 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.118304 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.119743 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:32.121365 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.121372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.121374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.121591 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.121802 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.121802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.121803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.121908 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.122195 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:32.122202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:32.122350 (+ 148us) client_negotiation.cc:769] Sending connection context
0808 21:06:32.122457 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 21:06:32.122610 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260808 21:06:32.123469 13948 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33233
I20260808 21:06:32.123564 13948 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:32.123746 13948 heartbeater.cc:511] Master 127.4.177.126:33233 requested a full tablet report, sending...
I20260808 21:06:32.124258 13740 ts_manager.cc:194] Registered new tserver with Master: 18c8ca724b17448fa030107f0a32198f (127.4.177.65:46425)
I20260808 21:06:32.125097 13740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:45511
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:32.149456 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:32.230062 13955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.230288 13955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.230326 13955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.231484 13955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.231552 13955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.231597 13955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:32.233014 13955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.233287 13955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.233515 13955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.238368 13963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.238399 13961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.238538 13955 server_base.cc:1034] running on GCE node
W20260808 21:06:32.238394 13960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.238756 13955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.238979 13955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.240144 13955 hybrid_clock.cc:648] HybridClock initialized: now 1786223192240148 us; error 39 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.242497 13955 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:32.242899 13955 webserver.cc:460] Webserver started at http://127.4.177.66:37713/ using document root <none> and password file <none>
I20260808 21:06:32.243168 13955 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.243247 13955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.243379 13955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:32.244421 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "42b8be7c55aa4dc5a8a8e8019e116091"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0f425297540a52963fb1cdb46e877573"
server_key_iv: "b72559406338fee06a0642f1e7267183"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.244760 13955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "42b8be7c55aa4dc5a8a8e8019e116091"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0f425297540a52963fb1cdb46e877573"
server_key_iv: "b72559406338fee06a0642f1e7267183"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.246230 13955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:32.246981 13970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.247179 13955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:32.247278 13955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "42b8be7c55aa4dc5a8a8e8019e116091"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0f425297540a52963fb1cdb46e877573"
server_key_iv: "b72559406338fee06a0642f1e7267183"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.247359 13955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.289287 13955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.291226 13955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.291386 13955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.291627 13955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:32.291944 13955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:32.291996 13955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.292040 13955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:32.292076 13955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.297695 13955 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:40989
I20260808 21:06:32.297773 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:40989 every 8 connection(s)
I20260808 21:06:32.298084 13955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:32.304611 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 13955
I20260808 21:06:32.304729 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:32.304927 4805 external_mini_cluster.cc:1428] Setting key 256878bd7e2078bc159be79e44ad5f59
I20260808 21:06:32.307094 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.298844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:50291 (local address 127.4.177.126:33233)
0808 21:06:32.298888 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.298890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.299080 (+ 190us) server_negotiation.cc:407] Connection header received
0808 21:06:32.299564 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.299565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.299575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.299642 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.300510 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.301001 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.301944 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.302152 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.302206 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.304240 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:32.304242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:32.304242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:32.304250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:32.305718 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.306196 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.306196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.306197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.306246 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.306511 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.306511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.306512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.306610 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:32.306657 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:32.306969 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 21:06:32.307030 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:06:32.307435 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.298800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:50291)
0808 21:06:32.298952 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.298994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.299483 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.299693 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.299696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.300249 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.300436 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.300440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.301075 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.301075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.301860 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.301862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.302223 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.302224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.302325 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.302778 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.302786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.304153 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:32.305788 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.305796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.305798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.306120 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.306302 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.306303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.306303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.306447 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.306667 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:32.306673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:32.306833 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 21:06:32.306941 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 21:06:32.307125 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260808 21:06:32.308115 14098 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33233
I20260808 21:06:32.308230 14098 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:32.308432 14098 heartbeater.cc:511] Master 127.4.177.126:33233 requested a full tablet report, sending...
I20260808 21:06:32.308781 13740 ts_manager.cc:194] Registered new tserver with Master: 42b8be7c55aa4dc5a8a8e8019e116091 (127.4.177.66:40989)
I20260808 21:06:32.309147 13740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:50291
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:32.331507 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:32.415748 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.415993 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.416033 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.417196 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.417253 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.417297 14104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:32.418720 14104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.418968 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.419238 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.423914 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.424048 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.423925 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.424115 14104 server_base.cc:1034] running on GCE node
I20260808 21:06:32.424331 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.424511 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.425668 14104 hybrid_clock.cc:648] HybridClock initialized: now 1786223192425661 us; error 40 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.428136 14104 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:32.428566 14104 webserver.cc:460] Webserver started at http://127.4.177.67:38017/ using document root <none> and password file <none>
I20260808 21:06:32.428800 14104 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.428864 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.429003 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:32.429997 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "9ea3a914094d4e0085eac3c4acfd3fec"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0292890668b22243e0cb56c001888e00"
server_key_iv: "6c642697390ff7a06386d6e13ec11c1b"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.430342 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea3a914094d4e0085eac3c4acfd3fec"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0292890668b22243e0cb56c001888e00"
server_key_iv: "6c642697390ff7a06386d6e13ec11c1b"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.431731 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 21:06:32.432422 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.432586 14104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:32.432683 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "9ea3a914094d4e0085eac3c4acfd3fec"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0292890668b22243e0cb56c001888e00"
server_key_iv: "6c642697390ff7a06386d6e13ec11c1b"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.432754 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.443063 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.444758 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.444940 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.445159 14104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:32.445434 14104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:32.445483 14104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.445521 14104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:32.445549 14104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.451521 14104 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:39007
I20260808 21:06:32.451556 14245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:39007 every 8 connection(s)
I20260808 21:06:32.451871 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:32.456032 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14104
I20260808 21:06:32.456118 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:32.456321 4805 external_mini_cluster.cc:1428] Setting key 28b8a32c42980869cae17cea2ba2a42a
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:32.460795 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.452523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:41841 (local address 127.4.177.126:33233)
0808 21:06:32.452573 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.452575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.452791 (+ 216us) server_negotiation.cc:407] Connection header received
0808 21:06:32.453219 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.453220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.453230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.453300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.454148 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.454649 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.455639 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.455818 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.455915 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.458205 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:32.458207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:32.458207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:32.458215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:32.459665 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.460050 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.460050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.460050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.460102 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:32.460305 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:32.460306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:32.460306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:32.460402 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:32.460450 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:32.460673 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 21:06:32.460731 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:06:32.460960 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.452509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:41841)
0808 21:06:32.452660 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.452693 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.453142 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.453348 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.453351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.453887 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.454067 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.454071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.454728 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.454729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.455545 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.455547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.455915 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.455916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.456011 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.456560 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.456570 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.458113 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:32.459739 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.459746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.459748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.459980 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.460155 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:32.460156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:32.460156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:32.460253 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:32.460451 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:32.460455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:32.460568 (+ 113us) client_negotiation.cc:769] Sending connection context
0808 21:06:32.460650 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 21:06:32.460765 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260808 21:06:32.461463 14246 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33233
I20260808 21:06:32.461570 14246 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:32.461799 14246 heartbeater.cc:511] Master 127.4.177.126:33233 requested a full tablet report, sending...
I20260808 21:06:32.462239 13740 ts_manager.cc:194] Registered new tserver with Master: 9ea3a914094d4e0085eac3c4acfd3fec (127.4.177.67:39007)
I20260808 21:06:32.462622 13740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:41841
I20260808 21:06:32.468017 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
Entry for principal oryx/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab.
WARNING: no policy specified for oryx/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
Entry for principal oryx/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab.
WARNING: no policy specified for oryx/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
Entry for principal oryx/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab.
I20260808 21:06:32.539682 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13808
I20260808 21:06:32.546149 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13955
I20260808 21:06:32.550861 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14104
I20260808 21:06:32.556255 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 13708
I20260808 21:06:32.562719 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33233
--webserver_interface=127.4.177.126
--webserver_port=44829
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:33233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:32.645852 14258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.646064 14258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.646086 14258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.647428 14258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:32.647469 14258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.647485 14258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.647511 14258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:32.647539 14258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:32.649010 14258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:33233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=44829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.649260 14258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.649479 14258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.654110 14267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.654134 14264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.654130 14265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.654378 14258 server_base.cc:1034] running on GCE node
I20260808 21:06:32.654632 14258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.654914 14258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.656067 14258 hybrid_clock.cc:648] HybridClock initialized: now 1786223192656060 us; error 36 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.658497 14258 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:32.658901 14258 webserver.cc:460] Webserver started at http://127.4.177.126:44829/ using document root <none> and password file <none>
I20260808 21:06:32.659118 14258 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.659233 14258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.660625 14258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:32.661259 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.661422 14258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:32.661518 14258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "e081182435c54154974819e9f61c6da2"
format_stamp: "Formatted at 2026-08-08 21:06:31 on dist-test-slave-chsk"
server_key: "87f6021c0023ac23255a4d82ec3d1077"
server_key_iv: "24b32f755cced3bedd65a7606c66caf0"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.661849 14258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.676133 14258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.677874 14258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.678020 14258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.682049 14258 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:33233
I20260808 21:06:32.682099 14332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:33233 every 8 connection(s)
I20260808 21:06:32.682492 14258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:32.684058 14333 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:06:32.684684 14333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Bootstrap starting.
I20260808 21:06:32.687721 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14258
I20260808 21:06:32.688002 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:46425
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:33233
--webserver_port=43099
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:06:32.694142 14333 log.cc:824] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:32.695861 14333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:06:32.696156 14333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Bootstrap complete.
I20260808 21:06:32.698455 14333 raft_consensus.cc:348] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:32.698683 14333 raft_consensus.cc:729] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e081182435c54154974819e9f61c6da2, State: Initialized, Role: FOLLOWER
I20260808 21:06:32.698848 14333 consensus_queue.cc:260] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:32.698917 14333 raft_consensus.cc:388] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:32.698974 14333 raft_consensus.cc:482] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:32.699023 14333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:06:32.700062 14333 raft_consensus.cc:504] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:32.700172 14333 leader_election.cc:302] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e081182435c54154974819e9f61c6da2; no voters:
I20260808 21:06:32.700335 14333 leader_election.cc:288] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 21:06:32.700387 14338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:06:32.700558 14338 raft_consensus.cc:686] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [term 2 LEADER]: Becoming Leader. State: Replica: e081182435c54154974819e9f61c6da2, State: Running, Role: LEADER
I20260808 21:06:32.700620 14333 sys_catalog.cc:564] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:32.700709 14338 consensus_queue.cc:237] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } }
I20260808 21:06:32.700966 14338 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e081182435c54154974819e9f61c6da2. Latest consensus state: current_term: 2 leader_uuid: "e081182435c54154974819e9f61c6da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } } }
I20260808 21:06:32.700971 14339 sys_catalog.cc:455] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e081182435c54154974819e9f61c6da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e081182435c54154974819e9f61c6da2" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33233 } } }
I20260808 21:06:32.701161 14338 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:32.701170 14339 sys_catalog.cc:458] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:32.701802 14346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:32.702376 14346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:32.702509 14346 catalog_manager.cc:1259] Loaded cluster ID: 3db05f250937484f93cb358e14adf7db
I20260808 21:06:32.702548 14346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:32.703749 14346 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:32.703922 14346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e081182435c54154974819e9f61c6da2: Loaded TSK: 0
I20260808 21:06:32.704145 14346 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 21:06:32.780841 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.781065 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.781112 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.782291 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.782353 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.782399 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:32.783946 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.65.keytab
--principal=oryx/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:46425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=43099
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.784204 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.784430 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.789351 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.789438 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.789331 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.790035 14335 server_base.cc:1034] running on GCE node
I20260808 21:06:32.790248 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.790493 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.791641 14335 hybrid_clock.cc:648] HybridClock initialized: now 1786223192791620 us; error 37 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.794379 14335 init.cc:377] Logged in from keytab as oryx/127.4.177.65@KRBTEST.COM (short username oryx)
I20260808 21:06:32.794930 14335 webserver.cc:460] Webserver started at http://127.4.177.65:43099/ using document root <none> and password file <none>
I20260808 21:06:32.795274 14335 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.795367 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.797192 14335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:32.797886 14374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.798089 14335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:32.798199 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "18c8ca724b17448fa030107f0a32198f"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0477f1b7e882b20e78f451b9c7716802"
server_key_iv: "c0431af1f9d7e287c05502bf032e491c"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.798625 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.812383 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.814999 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.815223 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.815488 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:32.815834 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:32.815883 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.815914 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:32.815940 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.823608 14335 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:46425
I20260808 21:06:32.823647 14497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:46425 every 8 connection(s)
I20260808 21:06:32.824079 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:32.833870 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.824858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:39513)
0808 21:06:32.825066 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.825117 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.825728 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.826163 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.826169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.826929 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.827254 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.827261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.828600 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.828601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.829884 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.829887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.830507 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.830508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.830643 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.831275 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.831282 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.833644 (+ 2362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
W20260808 21:06:32.833976 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.825021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:39513 (local address 127.4.177.126:33233)
0808 21:06:32.825185 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.825198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.825226 (+ 28us) server_negotiation.cc:407] Connection header received
0808 21:06:32.825924 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.825929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.825980 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.826078 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.827361 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.828488 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.829972 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.830390 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.830531 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.833790 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:39513: BlockingRecv error: recv got EOF from 127.4.177.65:39513 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260808 21:06:32.834146 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14335
I20260808 21:06:32.834477 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:40989
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:33233
--webserver_port=37713
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:32.834489 14498 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:33233 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.177.126:33233: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:32.840183 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.834848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:58163)
0808 21:06:32.834987 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.834995 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.835056 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.835209 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.835210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.835339 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.835384 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.835386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.836017 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.836018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.836844 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.836847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.837329 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.837331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.837407 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.837890 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.837896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.840103 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260808 21:06:32.840303 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.834889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:58163 (local address 127.4.177.126:33233)
0808 21:06:32.834926 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.834928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.835041 (+ 113us) server_negotiation.cc:407] Connection header received
0808 21:06:32.835122 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.835123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.835156 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.835209 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.835465 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.835933 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.836947 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.837240 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.837322 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.840220 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:58163: BlockingRecv error: recv got EOF from 127.4.177.65:58163 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:32.845851 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.840671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:41791)
0808 21:06:32.840716 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.840726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.840761 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.840910 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.840911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.841055 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.841090 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.841091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.842028 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.842029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.842779 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.842781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.843311 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.843312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.843369 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.843836 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.843841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.845785 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 21:06:32.846056 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.840708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:41791 (local address 127.4.177.126:33233)
0808 21:06:32.840743 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.840745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.840755 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:32.840817 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.840817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.840828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.840895 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.841172 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.841923 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.842873 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.843233 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.843312 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.845972 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:41791: BlockingRecv error: recv got EOF from 127.4.177.65:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 21:06:32.846216 14498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:06:32.919415 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:32.919620 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:32.919642 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:32.920804 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:32.920843 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:32.920869 14502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:32.922286 14502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.66.keytab
--principal=oryx/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:40989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=37713
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:32.922467 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:32.922652 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:32.927533 14512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.927541 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:32.927573 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:32.927896 14502 server_base.cc:1034] running on GCE node
I20260808 21:06:32.928120 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:32.928365 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:32.929527 14502 hybrid_clock.cc:648] HybridClock initialized: now 1786223192929514 us; error 26 us; skew 500 ppm
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223192, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:32.932158 14502 init.cc:377] Logged in from keytab as oryx/127.4.177.66@KRBTEST.COM (short username oryx)
I20260808 21:06:32.932627 14502 webserver.cc:460] Webserver started at http://127.4.177.66:37713/ using document root <none> and password file <none>
I20260808 21:06:32.932816 14502 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:32.932873 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:32.934194 14502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:32.934779 14519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.934974 14502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:32.935086 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "42b8be7c55aa4dc5a8a8e8019e116091"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0f425297540a52963fb1cdb46e877573"
server_key_iv: "b72559406338fee06a0642f1e7267183"
server_key_version: "encryptionkey@0"
I20260808 21:06:32.935464 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:32.952687 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:32.954608 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:32.954772 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:32.955014 14502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:32.955379 14502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:32.955431 14502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.955475 14502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:32.955516 14502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:32.961414 14502 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:40989
I20260808 21:06:32.961501 14643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:40989 every 8 connection(s)
I20260808 21:06:32.961818 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 21:06:32.969084 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.962548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:52279 (local address 127.4.177.126:33233)
0808 21:06:32.962598 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.962601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.962850 (+ 249us) server_negotiation.cc:407] Connection header received
0808 21:06:32.963394 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.963395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.963411 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.963464 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.964360 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.964892 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.965973 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.966155 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.966207 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.969025 (+ 2818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:52279: BlockingRecv error: recv got EOF from 127.4.177.66:52279 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:06:32.969146 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.962546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:52279)
0808 21:06:32.962733 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.962767 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.963303 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.963532 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.963543 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.964098 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.964286 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.964291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.964988 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.964989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.965894 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.965897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.966222 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.966223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.966317 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.966778 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.966786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.968953 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
W20260808 21:06:32.969358 14644 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:33233 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.177.126:33233: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
I20260808 21:06:32.971026 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14502
I20260808 21:06:32.971364 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:39007
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:33233
--webserver_port=38017
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:32.973716 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.969615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:53355)
0808 21:06:32.969648 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.969658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.969695 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.969811 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.969811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.969918 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.969960 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.969961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.970533 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.970534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.971023 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.971024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.971415 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.971416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.971481 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.971903 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.971908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.973670 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260808 21:06:32.973845 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.969655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:53355 (local address 127.4.177.126:33233)
0808 21:06:32.969708 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.969710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.969724 (+ 14us) server_negotiation.cc:407] Connection header received
0808 21:06:32.969743 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.969743 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.969751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.969801 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.970015 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.970456 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.971116 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.971348 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.971417 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.973768 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:53355: BlockingRecv error: recv got EOF from 127.4.177.66:53355 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:32 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:32.978638 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:32.974156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:37543)
0808 21:06:32.974260 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:32.974273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:32.974324 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:32.974449 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:32.974450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:32.974602 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:32.974638 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.974639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.975229 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.975231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.976105 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:32.976108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.976429 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.976431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:32.976493 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.976853 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:32.976859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:32.978557 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260808 21:06:32.978749 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:32.974213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:37543 (local address 127.4.177.126:33233)
0808 21:06:32.974251 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:32.974252 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:32.974309 (+ 57us) server_negotiation.cc:407] Connection header received
0808 21:06:32.974378 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:32.974379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:32.974386 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:32.974429 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:32.974703 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.975126 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.976187 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:32.976357 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:32.976411 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:32.978636 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:37543: BlockingRecv error: recv got EOF from 127.4.177.66:37543 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 21:06:32.978760 14644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:06:33.053730 14647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:33.053912 14647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:33.053936 14647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:33.055119 14647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:33.055218 14647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:33.055265 14647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:33.056757 14647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/krb5kdc/oryx_127.4.177.67.keytab
--principal=oryx/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:39007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=38017
--tserver_master_addrs=127.4.177.126:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:33.056968 14647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:33.057193 14647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:33.061992 14655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:33.062006 14652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:33.062006 14657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:33.062197 14647 server_base.cc:1034] running on GCE node
I20260808 21:06:33.062543 14647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:33.062806 14647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:33.063957 14647 hybrid_clock.cc:648] HybridClock initialized: now 1786223193063943 us; error 31 us; skew 500 ppm
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223193, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:33.066499 14647 init.cc:377] Logged in from keytab as oryx/127.4.177.67@KRBTEST.COM (short username oryx)
I20260808 21:06:33.066869 14647 webserver.cc:460] Webserver started at http://127.4.177.67:38017/ using document root <none> and password file <none>
I20260808 21:06:33.067129 14647 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:33.067236 14647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:33.068506 14647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:33.069051 14664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:33.069216 14647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:33.069299 14647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "9ea3a914094d4e0085eac3c4acfd3fec"
format_stamp: "Formatted at 2026-08-08 21:06:32 on dist-test-slave-chsk"
server_key: "0292890668b22243e0cb56c001888e00"
server_key_iv: "6c642697390ff7a06386d6e13ec11c1b"
server_key_version: "encryptionkey@0"
I20260808 21:06:33.069607 14647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:33.081075 14647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:33.082865 14647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:33.083014 14647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:33.083287 14647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:33.083590 14647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:33.083626 14647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:33.083665 14647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:33.083693 14647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:33.089421 14647 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:39007
I20260808 21:06:33.089469 14788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:39007 every 8 connection(s)
I20260808 21:06:33.089797 14647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
W20260808 21:06:33.096681 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:33.090491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:43689 (local address 127.4.177.126:33233)
0808 21:06:33.090577 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.090579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.090753 (+ 174us) server_negotiation.cc:407] Connection header received
0808 21:06:33.091262 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.091263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.091274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.091317 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.092175 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.092739 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.093683 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.093871 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.093919 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.096625 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:43689: BlockingRecv error: recv got EOF from 127.4.177.67:43689 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:06:33.096735 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.090457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:43689)
0808 21:06:33.090637 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:33.090671 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:33.091116 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:33.091369 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:33.091372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:33.091921 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:33.092104 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.092108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.092820 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.092821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.093614 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.093616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.093937 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.093938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.094030 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.094447 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:33.094454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:33.096544 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
W20260808 21:06:33.096952 14789 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:33233 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.177.126:33233: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
I20260808 21:06:33.097478 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14647
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:33.102269 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.097162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:42431)
0808 21:06:33.097223 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:33.097233 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:33.097270 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:33.097503 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:33.097504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:33.097647 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:33.097702 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.097704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.098514 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.098515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.099316 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.099319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.099741 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.099741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.099804 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.100215 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:33.100220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:33.102181 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 21:06:33.102293 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:33.097349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:42431 (local address 127.4.177.126:33233)
0808 21:06:33.097398 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.097400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.097407 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:06:33.097432 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.097433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.097440 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.097498 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.097795 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.098449 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.099382 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.099661 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.099735 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.102249 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:42431: BlockingRecv error: recv got EOF from 127.4.177.67:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 21:06:33.106269 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.097889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.4.177.126:33233)
0808 21:06:33.098226 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.098228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.098234 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:06:33.098253 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.098253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.098263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.098321 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.099026 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.099665 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.100850 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.101201 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.101265 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.102715 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:33.102719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:33.102719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:33.102731 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:33.104762 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:33.105167 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:33.105169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:33.105170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:33.105287 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:33.105484 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:33.105486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:33.105486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:33.105780 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:33.105927 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:33.106024 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 21:06:33.106160 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":69,"threads_started":1}
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:33.107468 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.102879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:34919)
0808 21:06:33.102928 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:33.102937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:33.102973 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:33.103080 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:33.103081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:33.103260 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:33.103292 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.103293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.104043 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.104044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.104836 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.104839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.105278 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.105279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.105340 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.105675 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:33.105680 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:33.107429 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 21:06:33.107556 14501 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:33.102939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:34919 (local address 127.4.177.126:33233)
0808 21:06:33.102982 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.102984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.102993 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:33.103014 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.103015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.103024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.103068 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.103386 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.103965 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.104907 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.105213 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.105261 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.107498 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:34919: BlockingRecv error: recv got EOF from 127.4.177.67:34919 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 21:06:33.107613 14789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:33.855809 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.846701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:49833)
0808 21:06:33.846849 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:33.846860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:33.846949 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:33.847166 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:33.847167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:33.847374 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:33.847435 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.847437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.848068 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.848068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.848610 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.848611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.848935 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.848936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.848974 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.853806 (+ 4832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:33.853813 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:33.855736 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
W20260808 21:06:33.855962 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:33.846768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:49833 (local address 127.4.177.126:33233)
0808 21:06:33.846877 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.846879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.847002 (+ 123us) server_negotiation.cc:407] Connection header received
0808 21:06:33.847028 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.847028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.847040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.847131 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.847499 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.847997 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.848679 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.848871 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.848928 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.855861 (+ 6933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:49833: BlockingRecv error: recv got EOF from 127.4.177.65:49833 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:33 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:33.983536 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:33.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:57285)
0808 21:06:33.979292 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:33.979304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:33.979369 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:33.979493 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:33.979493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:33.979675 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:33.979725 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.979726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.980349 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.980350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.980864 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:33.980865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.981169 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.981169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:33.981215 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.981660 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:33.981665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:33.983425 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260808 21:06:33.983570 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:33.979214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:57285 (local address 127.4.177.126:33233)
0808 21:06:33.979290 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:33.979292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:33.979367 (+ 75us) server_negotiation.cc:407] Connection header received
0808 21:06:33.979423 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:33.979424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:33.979435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:33.979475 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:33.979799 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.980285 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.980927 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:33.981111 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:33.981160 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:33.983504 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:57285: BlockingRecv error: recv got EOF from 127.4.177.66:57285 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:34.112464 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:34.107965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:41187)
0808 21:06:34.108102 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:34.108118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:34.108194 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:34.108324 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:34.108324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:34.108491 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:34.108540 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.108542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.109144 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.109145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.109677 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.109678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.110065 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.110066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.110119 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.110566 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:34.110570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:34.112351 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
W20260808 21:06:34.112498 14797 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:34.108054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:41187 (local address 127.4.177.126:33233)
0808 21:06:34.108129 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:34.108131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:34.108177 (+ 46us) server_negotiation.cc:407] Connection header received
0808 21:06:34.108242 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:34.108243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:34.108255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:34.108307 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:34.108603 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.109082 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.109742 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.109917 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.109967 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.112429 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:41187: BlockingRecv error: recv got EOF from 127.4.177.67:41187 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:34.860684 14800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:34.856274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:45995)
0808 21:06:34.856420 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:34.856431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:34.856480 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:34.856614 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:34.856615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:34.856776 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:34.856827 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.856829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.857467 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.857468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.857988 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.857989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.858335 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.858336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.858374 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.858809 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:34.858813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:34.860617 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260808 21:06:34.860771 14801 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:34.856362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:45995 (local address 127.4.177.126:33233)
0808 21:06:34.856487 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:34.856489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:34.856498 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:34.856522 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:34.856523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:34.856532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:34.856599 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:34.856890 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.857399 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.858058 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.858269 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.858323 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.860702 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:45995: BlockingRecv error: recv got EOF from 127.4.177.65:45995 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:34 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:34.988996 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:34.984170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:33595)
0808 21:06:34.984439 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:34.984468 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:34.984542 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:34.984729 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:34.984730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:34.984975 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:34.985037 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.985040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.985704 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.985705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.986220 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:34.986222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.986563 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.986564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:34.986634 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.987087 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:34.987091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:34.988938 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
W20260808 21:06:34.989161 14801 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:34.984300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:33595 (local address 127.4.177.126:33233)
0808 21:06:34.984358 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:34.984360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:34.984553 (+ 193us) server_negotiation.cc:407] Connection header received
0808 21:06:34.984614 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:34.984615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:34.984630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:34.984712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:34.985137 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.985621 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.986311 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:34.986498 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:34.986555 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:34.989064 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:33595: BlockingRecv error: recv got EOF from 127.4.177.66:33595 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:35.117619 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:35.113022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:56481)
0808 21:06:35.113161 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:35.113174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:35.113242 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:35.113374 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:35.113375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:35.113542 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:35.113617 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.113619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.114407 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.114408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.114984 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.114987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.115385 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.115386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.115426 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.115837 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:35.115841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:35.117561 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
W20260808 21:06:35.117753 14801 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:35.113175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:56481 (local address 127.4.177.126:33233)
0808 21:06:35.113245 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:35.113247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:35.113284 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:06:35.113306 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:35.113307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:35.113318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:35.113358 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:35.113702 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.114329 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.115058 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.115316 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.115373 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.117652 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:56481: BlockingRecv error: recv got EOF from 127.4.177.67:56481 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:35.866108 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:35.861245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:36181)
0808 21:06:35.861395 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:35.861406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:35.861459 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:35.861675 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:35.861676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:35.861842 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:35.861892 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.861894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.862549 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.862550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.863074 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.863076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.863468 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.863469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.863512 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.864048 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:35.864054 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:35.866047 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260808 21:06:35.866226 14805 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:35.861392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:36181 (local address 127.4.177.126:33233)
0808 21:06:35.861529 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:35.861531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:35.861541 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:35.861574 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:35.861575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:35.861585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:35.861662 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:35.861964 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.862478 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.863209 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.863404 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.863458 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.866136 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:36181: BlockingRecv error: recv got EOF from 127.4.177.65:36181 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:35 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:35.994443 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:35.989590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:57967)
0808 21:06:35.989747 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:35.989759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:35.989825 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:35.989967 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:35.989968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:35.990134 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:35.990196 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.990198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.990840 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.990840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.991398 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:35.991399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.991728 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.991728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:35.991768 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.992278 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:35.992283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:35.994341 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
W20260808 21:06:35.994491 14805 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:35.989666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:57967 (local address 127.4.177.126:33233)
0808 21:06:35.989755 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:35.989757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:35.989827 (+ 70us) server_negotiation.cc:407] Connection header received
0808 21:06:35.989877 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:35.989878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:35.989889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:35.989954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:35.990275 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.990768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.991473 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:35.991661 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:35.991717 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:35.994426 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:57967: BlockingRecv error: recv got EOF from 127.4.177.66:57967 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:36.122834 14807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:36.118103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:51603)
0808 21:06:36.118287 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:36.118299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:36.118373 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:36.118553 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:36.118554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:36.118745 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:36.118793 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.118794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.119545 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.119546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.120115 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.120116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.120462 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.120463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.120529 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:36.120989 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:36.120994 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:36.122738 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
W20260808 21:06:36.122884 14805 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:36.118204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:51603 (local address 127.4.177.126:33233)
0808 21:06:36.118283 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:36.118285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:36.118352 (+ 67us) server_negotiation.cc:407] Connection header received
0808 21:06:36.118434 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:36.118435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:36.118450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:36.118532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:36.118873 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.119450 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.120200 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.120390 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.120464 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:36.122816 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:51603: BlockingRecv error: recv got EOF from 127.4.177.67:51603 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:36.871120 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:36.866573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:36755)
0808 21:06:36.866762 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:36.866774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:36.866831 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:36.866954 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:36.866955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:36.867160 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:36.867213 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.867215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.867847 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.867847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.868373 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.868375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.868691 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.868691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.868729 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:36.869303 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:36.869308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:36.871045 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
W20260808 21:06:36.871297 14809 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:36.866618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:36755 (local address 127.4.177.126:33233)
0808 21:06:36.866745 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:36.866747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:36.866830 (+ 83us) server_negotiation.cc:407] Connection header received
0808 21:06:36.866884 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:36.866885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:36.866897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:36.866943 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:36.867276 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.867780 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.868439 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.868631 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.868680 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:36.871195 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:36755: BlockingRecv error: recv got EOF from 127.4.177.65:36755 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":49,"threads_started":1}
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:36 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:37.000116 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:36.995054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:38801)
0808 21:06:36.995351 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:36.995365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:36.995444 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:36.995578 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:36.995578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:36.995765 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:36.995813 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.995815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.996574 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.996576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.997159 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:36.997162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.997500 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.997500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:36.997542 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:36.998016 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:36.998021 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:37.000037 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
W20260808 21:06:37.000207 14809 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:36.995195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:38801 (local address 127.4.177.126:33233)
0808 21:06:36.995369 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:36.995372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:36.995424 (+ 52us) server_negotiation.cc:407] Connection header received
0808 21:06:36.995505 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:36.995506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:36.995519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:36.995572 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:36.995896 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.996475 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.997245 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:36.997430 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:36.997486 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:37.000144 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:38801: BlockingRecv error: recv got EOF from 127.4.177.66:38801 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Aug 08 21:06:37 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:37 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:37.128038 14811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:37.123434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:57337)
0808 21:06:37.123613 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:37.123623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:37.123683 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:37.123828 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:37.123829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:37.124001 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:37.124052 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:37.124053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.124686 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.124687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:37.125211 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:37.125212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.125538 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.125538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:37.125577 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:37.126034 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:37.126040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:37.127980 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
W20260808 21:06:37.128155 14809 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:37.123537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:57337 (local address 127.4.177.126:33233)
0808 21:06:37.123588 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:37.123590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:37.123673 (+ 83us) server_negotiation.cc:407] Connection header received
0808 21:06:37.123734 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:37.123735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:37.123746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:37.123810 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:37.124122 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.124611 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.125297 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.125479 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.125529 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:37.128069 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:57337: BlockingRecv error: recv got EOF from 127.4.177.67:57337 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Aug 08 21:06:37 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:37 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:37.876488 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:37.871627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:41687)
0808 21:06:37.871784 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:37.871799 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:37.871869 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:37.872001 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:37.872001 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:37.872196 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:37.872248 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:37.872249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.872965 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.872966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:37.873535 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:37.873537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.874039 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.874040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:37.874098 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:37.874547 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:37.874553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:37.876425 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
W20260808 21:06:37.876591 14813 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:37.871658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:41687 (local address 127.4.177.126:33233)
0808 21:06:37.871811 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:37.871813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:37.871853 (+ 40us) server_negotiation.cc:407] Connection header received
0808 21:06:37.871913 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:37.871914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:37.871924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:37.871986 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:37.872307 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.872892 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.873610 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:37.873949 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:37.874026 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:37.876520 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:41687: BlockingRecv error: recv got EOF from 127.4.177.65:41687 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:38.005236 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:38.000575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:33089)
0808 21:06:38.000705 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:38.000717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:38.000778 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:38.000905 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:38.000905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:38.001147 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:38.001208 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.001215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.001929 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.001929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.002523 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.002525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.002861 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.002862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.002902 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.003368 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:38.003374 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:38.005176 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260808 21:06:38.005344 14813 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:38.000685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:33089 (local address 127.4.177.126:33233)
0808 21:06:38.000732 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:38.000734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:38.000762 (+ 28us) server_negotiation.cc:407] Connection header received
0808 21:06:38.000830 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:38.000831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:38.000843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:38.000901 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:38.001297 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.001834 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.002595 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.002792 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.002854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.005271 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:33089: BlockingRecv error: recv got EOF from 127.4.177.66:33089 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:38.133726 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:38.128677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:35377)
0808 21:06:38.128880 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:38.128891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:38.128970 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:38.129114 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:38.129115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:38.129304 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:38.129375 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.129377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.130316 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.130317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.130850 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.130852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.131283 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.131284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.131323 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.131811 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:38.131816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:38.133660 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
W20260808 21:06:38.133819 14813 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:38.128792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:35377 (local address 127.4.177.126:33233)
0808 21:06:38.128864 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:38.128867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:38.128948 (+ 81us) server_negotiation.cc:407] Connection header received
0808 21:06:38.129036 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:38.129037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:38.129051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:38.129099 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:38.129489 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.130235 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.130914 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.131110 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.131203 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.133754 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:35377: BlockingRecv error: recv got EOF from 127.4.177.67:35377 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:38 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:38.891036 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:38.876999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:57593)
0808 21:06:38.877145 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:38.877155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:38.877212 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:38.877379 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:38.877379 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:38.877546 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:38.877599 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.877600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.878220 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.878220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.878742 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:38.878744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.879050 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.879051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:38.879091 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.889110 (+ 10019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:38.889116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:38.890974 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
W20260808 21:06:38.891201 14817 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:38.877053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:57593 (local address 127.4.177.126:33233)
0808 21:06:38.877208 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:38.877211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:38.877221 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:38.877274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:38.877274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:38.877286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:38.877362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:38.877658 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.878152 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.878805 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:38.878988 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:38.879038 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:38.891090 (+ 12052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:57593: BlockingRecv error: recv got EOF from 127.4.177.65:57593 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:39.010169 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:39.005680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:37139)
0808 21:06:39.005846 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:39.005856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:39.005913 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:39.006050 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:39.006051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:39.006224 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:39.006277 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.006279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.006895 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.006896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.007464 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.007465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.007778 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.007779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.007820 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.008274 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:39.008279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:39.010097 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
W20260808 21:06:39.010336 14817 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:39.005753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:37139 (local address 127.4.177.126:33233)
0808 21:06:39.005814 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:39.005816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:39.005912 (+ 96us) server_negotiation.cc:407] Connection header received
0808 21:06:39.005976 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:39.005977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:39.005989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:39.006031 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:39.006338 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.006830 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.007533 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.007718 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.007769 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.010249 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:37139: BlockingRecv error: recv got EOF from 127.4.177.66:37139 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:39.139092 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:39.134331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:36799)
0808 21:06:39.134524 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:39.134542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:39.134622 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:39.134768 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:39.134769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:39.134951 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:39.134999 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.135001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.135725 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.135726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.136243 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.136245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.136617 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.136618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.136677 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.137174 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:39.137179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:39.139014 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":110,"threads_started":1}
W20260808 21:06:39.139231 14817 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:39.134429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:36799 (local address 127.4.177.126:33233)
0808 21:06:39.134507 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:39.134509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:39.134601 (+ 92us) server_negotiation.cc:407] Connection header received
0808 21:06:39.134692 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:39.134693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:39.134707 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:39.134763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:39.135084 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.135641 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.136329 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.136535 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.136606 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.139103 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:36799: BlockingRecv error: recv got EOF from 127.4.177.67:36799 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:39 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:39.896102 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:39.891610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:34061)
0808 21:06:39.891739 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:39.891756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:39.891806 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:39.891967 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:39.891968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:39.892153 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:39.892206 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.892208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.892821 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.892821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.893375 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:39.893377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.893685 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.893686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:39.893724 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.894234 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:39.894239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:39.896003 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
W20260808 21:06:39.896219 14822 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:39.891690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:34061 (local address 127.4.177.126:33233)
0808 21:06:39.891837 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:39.891839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:39.891848 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:39.891870 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:39.891871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:39.891881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:39.891952 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:39.892271 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.892758 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.893443 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:39.893625 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:39.893673 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:39.896132 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:34061: BlockingRecv error: recv got EOF from 127.4.177.65:34061 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:40.015609 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:40.010734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:43223)
0808 21:06:40.010917 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:40.010929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:40.011016 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:40.011265 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:40.011266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:40.011441 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:40.011493 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.011495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.012170 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.012172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.012691 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.012692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.013083 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.013084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.013148 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.013597 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:40.013602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:40.015512 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
W20260808 21:06:40.015657 14822 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:40.010824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:43223 (local address 127.4.177.126:33233)
0808 21:06:40.010913 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:40.010916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:40.010999 (+ 83us) server_negotiation.cc:407] Connection header received
0808 21:06:40.011095 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:40.011096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:40.011109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:40.011175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:40.011577 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.012080 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.012804 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.012999 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.013082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.015591 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:43223: BlockingRecv error: recv got EOF from 127.4.177.66:43223 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:40.144398 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:40.139617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:58729)
0808 21:06:40.139834 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:40.139847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:40.139916 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:40.140052 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:40.140053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:40.140232 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:40.140286 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.140288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.140961 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.140962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.141550 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.141552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.141932 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.141932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.141972 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.142497 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:40.142502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:40.144327 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":128,"threads_started":1}
W20260808 21:06:40.144518 14822 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:40.139681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:58729 (local address 127.4.177.126:33233)
0808 21:06:40.139756 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:40.139758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:40.139917 (+ 159us) server_negotiation.cc:407] Connection header received
0808 21:06:40.139965 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:40.139966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:40.139980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:40.140034 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:40.140380 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.140881 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.141654 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.141858 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.141925 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.144431 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:58729: BlockingRecv error: recv got EOF from 127.4.177.67:58729 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:40 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:40.901371 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:40.896602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:35869)
0808 21:06:40.896780 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:40.896795 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:40.896860 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:40.897110 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:40.897111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:40.897286 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:40.897336 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.897337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.897979 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.897979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.898501 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:40.898502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.898851 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.898851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:40.898891 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.899402 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:40.899406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:40.901246 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
W20260808 21:06:40.901453 14826 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:40.896850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:35869 (local address 127.4.177.126:33233)
0808 21:06:40.896970 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:40.896972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:40.896981 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:40.897015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:40.897015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:40.897026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:40.897086 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:40.897397 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.897916 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.898582 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:40.898787 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:40.898842 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:40.901371 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:35869: BlockingRecv error: recv got EOF from 127.4.177.65:35869 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:41.021472 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:41.016231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:37281)
0808 21:06:41.016399 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:41.016410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:41.016471 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:41.016654 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:41.016655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:41.016836 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:41.016888 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.016890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.017755 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.017756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.018307 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.018309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.018684 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.018685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.018732 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.019371 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:41.019376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:41.021408 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
W20260808 21:06:41.021610 14826 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:41.016367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:37281 (local address 127.4.177.126:33233)
0808 21:06:41.016436 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:41.016438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:41.016455 (+ 17us) server_negotiation.cc:407] Connection header received
0808 21:06:41.016530 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:41.016530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:41.016543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:41.016591 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:41.016992 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.017663 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.018390 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.018602 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.018662 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.021529 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:37281: BlockingRecv error: recv got EOF from 127.4.177.66:37281 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:41.149418 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:41.144955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:58689)
0808 21:06:41.145107 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:41.145126 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:41.145186 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:41.145305 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:41.145306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:41.145483 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:41.145531 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.145532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.146139 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.146140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.146671 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.146673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.147006 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.147007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.147043 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.147513 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:41.147518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:41.149348 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260808 21:06:41.149582 14826 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:41.145021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:58689 (local address 127.4.177.126:33233)
0808 21:06:41.145100 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:41.145102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:41.145190 (+ 88us) server_negotiation.cc:407] Connection header received
0808 21:06:41.145229 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:41.145230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:41.145241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:41.145292 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:41.145595 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.146077 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.146742 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.146944 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.146996 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.149475 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:58689: BlockingRecv error: recv got EOF from 127.4.177.67:58689 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:41 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:41.906953 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:41.901951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:39669)
0808 21:06:41.902105 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:41.902120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:41.902189 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:41.902383 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:41.902384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:41.902564 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:41.902647 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.902649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.903516 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.903517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.904187 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:41.904189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.904523 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.904524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:41.904565 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.905098 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:41.905103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:41.906894 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
W20260808 21:06:41.907068 14830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:41.902032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:39669 (local address 127.4.177.126:33233)
0808 21:06:41.902172 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:41.902174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:41.902185 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:41.902258 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:41.902258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:41.902269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:41.902330 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:41.902724 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.903432 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.904268 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:41.904459 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:41.904512 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:41.906985 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:39669: BlockingRecv error: recv got EOF from 127.4.177.65:39669 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:42.030738 14831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:42.021986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:58611)
0808 21:06:42.022194 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:42.022205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:42.022270 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:42.022446 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:42.022447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:42.022623 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:42.022673 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.022675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.023368 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.023369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.024048 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.024051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.024376 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.024376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.024417 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.028765 (+ 4348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:42.028770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:42.030667 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":134,"threads_started":1}
W20260808 21:06:42.030838 14830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:42.022098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:58611 (local address 127.4.177.126:33233)
0808 21:06:42.022181 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:42.022188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:42.022270 (+ 82us) server_negotiation.cc:407] Connection header received
0808 21:06:42.022351 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:42.022351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:42.022368 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:42.022431 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:42.022741 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.023310 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.024123 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.024310 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.024364 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.030769 (+ 6405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:58611: BlockingRecv error: recv got EOF from 127.4.177.66:58611 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:42.155807 14832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:42.149932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:52723)
0808 21:06:42.150045 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:42.150056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:42.150126 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:42.150302 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:42.150303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:42.150527 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:42.150596 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.150599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.151726 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.151727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.152719 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.152721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.153280 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.153281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.153350 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.153873 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:42.153878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:42.155749 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260808 21:06:42.156019 14830 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:42.149987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:52723 (local address 127.4.177.126:33233)
0808 21:06:42.150033 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:42.150035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:42.150111 (+ 76us) server_negotiation.cc:407] Connection header received
0808 21:06:42.150177 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:42.150178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:42.150189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:42.150275 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:42.150702 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.151624 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.152825 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.153167 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.153247 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.155868 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:52723: BlockingRecv error: recv got EOF from 127.4.177.67:52723 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:42 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:42.912611 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:42.907642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:38653)
0808 21:06:42.907780 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:42.907794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:42.907854 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:42.908058 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:42.908059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:42.908262 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:42.908315 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.908317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.909022 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.909023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.909582 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:42.909584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.909974 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.909975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:42.910045 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.910601 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:42.910606 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:42.912508 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260808 21:06:42.912688 14834 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:42.907701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:38653 (local address 127.4.177.126:33233)
0808 21:06:42.907822 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:42.907824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:42.907837 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:06:42.907942 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:42.907943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:42.907958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:42.908035 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:42.908403 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.908921 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.909677 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:42.909881 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:42.909971 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:42.912604 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:38653: BlockingRecv error: recv got EOF from 127.4.177.65:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:43.036676 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:43.031389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:57021)
0808 21:06:43.031576 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:43.031589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:43.031659 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:43.031857 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:43.031859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:43.032039 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:43.032090 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.032091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.032804 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.032805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.033353 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.033354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.033948 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.033949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.034005 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.034502 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:43.034507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:43.036607 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
W20260808 21:06:43.036870 14834 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:43.031511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:57021 (local address 127.4.177.126:33233)
0808 21:06:43.031589 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:43.031591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:43.031707 (+ 116us) server_negotiation.cc:407] Connection header received
0808 21:06:43.031741 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:43.031742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:43.031755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:43.031816 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:43.032186 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.032702 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.033517 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.033839 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.033926 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.036789 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:57021: BlockingRecv error: recv got EOF from 127.4.177.66:57021 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:43.160776 14836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:43.156332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:39483)
0808 21:06:43.156490 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:43.156501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:43.156550 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:43.156698 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:43.156698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:43.156872 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:43.156923 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.156924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.157581 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.157582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.158129 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.158130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.158452 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.158453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.158492 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.158930 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:43.158934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:43.160708 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
W20260808 21:06:43.160851 14834 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:43.156503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:39483 (local address 127.4.177.126:33233)
0808 21:06:43.156573 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:43.156577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:43.156589 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:43.156611 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:43.156612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:43.156623 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:43.156682 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:43.156989 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.157502 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.158211 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.158388 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.158441 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.160801 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:39483: BlockingRecv error: recv got EOF from 127.4.177.67:39483 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:43 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:43.920653 14837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:43.913096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:46343)
0808 21:06:43.913265 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:43.913276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:43.913336 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:43.913477 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:43.913478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:43.913654 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:43.913700 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.913702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.914343 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.914343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.914875 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:43.914876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.915292 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.915293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:43.915334 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.918809 (+ 3475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:43.918814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:43.920579 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
W20260808 21:06:43.920723 14838 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:43.913200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:46343 (local address 127.4.177.126:33233)
0808 21:06:43.913328 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:43.913329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:43.913338 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:43.913399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:43.913399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:43.913410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:43.913457 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:43.913765 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.914280 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.914979 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:43.915218 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:43.915292 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:43.920670 (+ 5378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:46343: BlockingRecv error: recv got EOF from 127.4.177.65:46343 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:44.042198 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:44.037387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:54321)
0808 21:06:44.037576 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:44.037586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:44.037646 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:44.037826 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:44.037826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:44.038003 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:44.038053 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.038054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.038682 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.038683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.039277 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.039280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.039637 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.039638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.039678 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.040130 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:44.040136 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:44.042130 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":121,"threads_started":1}
W20260808 21:06:44.042312 14838 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:44.037482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:54321 (local address 127.4.177.126:33233)
0808 21:06:44.037551 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:44.037553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:44.037651 (+ 98us) server_negotiation.cc:407] Connection header received
0808 21:06:44.037717 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:44.037718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:44.037732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:44.037813 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:44.038118 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.038609 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.039369 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.039568 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.039629 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.042215 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:54321: BlockingRecv error: recv got EOF from 127.4.177.66:54321 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:44.165933 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:44.161291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:32933)
0808 21:06:44.161481 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:44.161490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:44.161552 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:44.161684 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:44.161684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:44.161856 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:44.161900 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.161902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.162524 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.162525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.163057 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.163058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.163438 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.163438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.163479 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.164015 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:44.164019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:44.165864 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
W20260808 21:06:44.166044 14838 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:44.161378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:32933 (local address 127.4.177.126:33233)
0808 21:06:44.161437 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:44.161439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:44.161547 (+ 108us) server_negotiation.cc:407] Connection header received
0808 21:06:44.161596 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:44.161597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:44.161609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:44.161660 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:44.161962 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.162459 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.163131 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.163370 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.163425 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.165948 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:32933: BlockingRecv error: recv got EOF from 127.4.177.67:32933 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:44 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:44.925806 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:44.921170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:46051)
0808 21:06:44.921359 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:44.921370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:44.921428 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:44.921562 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:44.921563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:44.921742 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:44.921789 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.921790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.922550 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.922551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.923089 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:44.923091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.923496 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.923497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:44.923543 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.924021 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:44.924025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:44.925746 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":120,"threads_started":1}
W20260808 21:06:44.925972 14842 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:44.921264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:46051 (local address 127.4.177.126:33233)
0808 21:06:44.921436 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:44.921438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:44.921448 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:44.921472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:44.921473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:44.921484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:44.921560 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:44.921867 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.922462 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.923232 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:44.923426 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:44.923483 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:44.925888 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:46051: BlockingRecv error: recv got EOF from 127.4.177.65:46051 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":114,"threads_started":1}
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:45.047720 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:45.042693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:39927)
0808 21:06:45.042874 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:45.042884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:45.042972 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:45.043157 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:45.043157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:45.043353 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:45.043401 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.043403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.044082 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.044083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.044607 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.044609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.044978 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.044980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.045045 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.045549 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:45.045554 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:45.047612 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
W20260808 21:06:45.047761 14842 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:45.042805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:39927 (local address 127.4.177.126:33233)
0808 21:06:45.042869 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:45.042871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:45.042961 (+ 90us) server_negotiation.cc:407] Connection header received
0808 21:06:45.043036 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:45.043037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:45.043051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:45.043132 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:45.043487 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.043981 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.044703 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.044902 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.044980 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.047699 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:39927: BlockingRecv error: recv got EOF from 127.4.177.66:39927 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:45.171471 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:45.166461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:42071)
0808 21:06:45.166649 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:45.166662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:45.166766 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:45.166938 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:45.166939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:45.167192 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:45.167273 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.167275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.167956 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.167957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.168492 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.168494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.168864 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.168865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.168936 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.169498 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:45.169503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:45.171357 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
W20260808 21:06:45.171514 14842 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:45.166560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:42071 (local address 127.4.177.126:33233)
0808 21:06:45.166634 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:45.166637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:45.166737 (+ 100us) server_negotiation.cc:407] Connection header received
0808 21:06:45.166843 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:45.166844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:45.166861 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:45.166932 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:45.167356 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.167866 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.168589 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.168778 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.168853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.171448 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:42071: BlockingRecv error: recv got EOF from 127.4.177.67:42071 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:45 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:45.931298 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:45.926340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:50781)
0808 21:06:45.926522 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:45.926537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:45.926611 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:45.926759 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:45.926760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:45.926936 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:45.926983 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.926985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.927737 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.927738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.928308 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:45.928310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.928704 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.928706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:45.928756 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.929240 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:45.929245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:45.931213 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20260808 21:06:45.931396 14846 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:45.926411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:50781 (local address 127.4.177.126:33233)
0808 21:06:45.926601 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:45.926603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:45.926615 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:45.926673 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:45.926674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:45.926685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:45.926740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:45.927048 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.927654 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.928401 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:45.928611 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:45.928677 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:45.931304 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:50781: BlockingRecv error: recv got EOF from 127.4.177.65:50781 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":81,"threads_started":1}
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:46.052929 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:46.048275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:50347)
0808 21:06:46.048399 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:46.048412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:46.048492 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:46.048685 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:46.048685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:46.048935 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:46.049000 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.049002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.049652 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.049652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.050170 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.050171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.050522 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.050522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.050561 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.051045 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:46.051050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:46.052870 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20260808 21:06:46.053145 14846 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:46.048376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:50347 (local address 127.4.177.126:33233)
0808 21:06:46.048446 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:46.048448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:46.048459 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:46.048578 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:46.048579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:46.048592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:46.048656 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:46.049077 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.049568 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.050259 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.050450 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.050513 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.053039 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:50347: BlockingRecv error: recv got EOF from 127.4.177.66:50347 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:46.176867 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:46.172013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:42407)
0808 21:06:46.172175 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:46.172191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:46.172247 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:46.172378 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:46.172378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:46.172554 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:46.172602 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.172603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.173295 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.173295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.173841 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.173843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.174334 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.174335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.174387 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.174863 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:46.174867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:46.176807 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
W20260808 21:06:46.176991 14846 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:46.172078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:42407 (local address 127.4.177.126:33233)
0808 21:06:46.172151 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:46.172154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:46.172244 (+ 90us) server_negotiation.cc:407] Connection header received
0808 21:06:46.172298 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:46.172298 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:46.172310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:46.172381 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:46.172655 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.173210 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.173975 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.174233 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.174302 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.176892 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:42407: BlockingRecv error: recv got EOF from 127.4.177.67:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:46 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:46.936465 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:46.931779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:44293)
0808 21:06:46.931942 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:46.931960 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:46.932009 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:46.932162 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:46.932163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:46.932336 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:46.932386 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.932387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.933025 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.933026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.933532 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:46.933533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.933861 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.933861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:46.933901 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.934354 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:46.934359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:46.936404 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
W20260808 21:06:46.936596 14851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:46.931881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:44293 (local address 127.4.177.126:33233)
0808 21:06:46.932020 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:46.932022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:46.932035 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:06:46.932059 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:46.932060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:46.932070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:46.932146 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:46.932455 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.932956 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.933605 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:46.933792 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:46.933848 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:46.936505 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:44293: BlockingRecv error: recv got EOF from 127.4.177.65:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:47.058558 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:47.053474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:44157)
0808 21:06:47.053659 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:47.053675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:47.053774 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:47.053934 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:47.053935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:47.054124 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:47.054180 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.054182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.055113 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.055115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.055718 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.055720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.056113 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.056113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.056188 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.056738 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:47.056743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:47.058485 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
W20260808 21:06:47.058630 14851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:47.053587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:44157 (local address 127.4.177.126:33233)
0808 21:06:47.053677 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:47.053679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:47.053754 (+ 75us) server_negotiation.cc:407] Connection header received
0808 21:06:47.053855 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:47.053856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:47.053868 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:47.053928 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:47.054263 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.055040 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.055805 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.056021 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.056109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.058582 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:44157: BlockingRecv error: recv got EOF from 127.4.177.66:44157 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.67@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:47.181739 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:47.177326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.67:43645)
0808 21:06:47.177493 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:47.177505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:47.177567 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:47.177697 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:47.177699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:47.177873 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:47.177922 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.177923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.178540 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.178541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.179074 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.179075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.179461 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.179461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.179504 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.179943 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:47.179948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:47.181682 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
W20260808 21:06:47.181880 14851 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:47.177395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:43645 (local address 127.4.177.126:33233)
0808 21:06:47.177460 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:47.177462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:47.177569 (+ 107us) server_negotiation.cc:407] Connection header received
0808 21:06:47.177610 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:47.177611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:47.177624 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:47.177679 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:47.177983 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.178476 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.179183 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.179381 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.179438 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.181791 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.67:43645: BlockingRecv error: recv got EOF from 127.4.177.67:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:47 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.65@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:47.941859 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:47.937036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.65:57355)
0808 21:06:47.937208 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:47.937219 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:47.937266 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:47.937401 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:47.937402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:47.937567 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:47.937618 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.937620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.938271 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.938272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.938858 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:47.938860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.939269 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.939271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:47.939346 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.939786 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:47.939790 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:47.941785 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
W20260808 21:06:47.941968 14856 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:47.937121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:57355 (local address 127.4.177.126:33233)
0808 21:06:47.937247 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:47.937249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:47.937259 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:06:47.937312 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:47.937312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:47.937323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:47.937386 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:47.937687 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.938194 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.938953 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:47.939199 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:47.939262 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:47.941890 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.65:57355: BlockingRecv error: recv got EOF from 127.4.177.65:57355 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
Aug 08 21:06:48 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
Aug 08 21:06:48 dist-test-slave-chsk krb5kdc[13693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.177.66@KRBTEST.COM for oryx/127.4.177.126@KRBTEST.COM, Server not found in Kerberos database
I20260808 21:06:48.063645 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.059177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33233 (local address 127.4.177.66:51399)
0808 21:06:48.059311 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:48.059322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:48.059378 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:48.059510 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:48.059511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:48.059683 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:48.059733 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.059734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.060433 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.060434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.060946 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.060947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.061251 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.061251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.061289 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.061795 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:48.061800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:48.063586 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.177.126:33233: Server oryx/127.4.177.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260808 21:06:48.063750 14856 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:48.059268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:51399 (local address 127.4.177.126:33233)
0808 21:06:48.059326 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.059328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.059364 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:06:48.059426 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.059427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.059438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.059491 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.059825 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.060344 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.061012 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.061189 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.061239 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.063679 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.177.66:51399: BlockingRecv error: recv got EOF from 127.4.177.66:51399 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 21:06:48.108088 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14335
I20260808 21:06:48.114288 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14502
I20260808 21:06:48.120168 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14647
I20260808 21:06:48.125788 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14258
2026-08-08T21:06:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17571 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T21:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:48Z Disabled control of system clock
I20260808 21:06:48.157110 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43657
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 21:06:48.241513 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:48.241744 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:48.241767 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:48.243005 14861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:48.243049 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:48.243062 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:48.243075 14861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:48.243088 14861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:48.244541 14861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:43657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:48.244812 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:48.245100 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:48.250432 14867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.250473 14869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.250463 14866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.250721 14861 server_base.cc:1034] running on GCE node
I20260808 21:06:48.250977 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:48.251360 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:48.252516 14861 hybrid_clock.cc:648] HybridClock initialized: now 1786223208252498 us; error 31 us; skew 500 ppm
I20260808 21:06:48.253937 14861 webserver.cc:460] Webserver started at http://127.4.177.126:42509/ using document root <none> and password file <none>
I20260808 21:06:48.254212 14861 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:48.254264 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:48.254408 14861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:48.255599 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "9d4adfed69e5416a813da1b4664030f7"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "be13df85160fed79fa80c8c2008c278a"
server_key_iv: "3d5725a1191bf602ae378186a0135d78"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.255978 14861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "9d4adfed69e5416a813da1b4664030f7"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "be13df85160fed79fa80c8c2008c278a"
server_key_iv: "3d5725a1191bf602ae378186a0135d78"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.257424 14861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:48.258190 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.258386 14861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:06:48.258482 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "9d4adfed69e5416a813da1b4664030f7"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "be13df85160fed79fa80c8c2008c278a"
server_key_iv: "3d5725a1191bf602ae378186a0135d78"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.258564 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:48.280027 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:48.280408 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:48.280588 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:48.285020 14861 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:43657
I20260808 21:06:48.285100 14932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:43657 every 8 connection(s)
I20260808 21:06:48.285465 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:48.286158 14934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:48.288856 14934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7: Bootstrap starting.
I20260808 21:06:48.289508 14934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:48.289839 14934 log.cc:824] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:48.290567 14934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7: No bootstrap required, opened a new log
I20260808 21:06:48.292326 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14861
I20260808 21:06:48.292232 14934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } }
I20260808 21:06:48.292404 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:48.292402 14934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:48.292467 14934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d4adfed69e5416a813da1b4664030f7, State: Initialized, Role: FOLLOWER
I20260808 21:06:48.292647 4805 external_mini_cluster.cc:1428] Setting key 9439f5af3c25c753d0aae2e82aa60da0
I20260808 21:06:48.292651 14934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } }
I20260808 21:06:48.292763 14934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:48.292809 14934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:48.292873 14934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:48.293715 14934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } }
I20260808 21:06:48.293833 14934 leader_election.cc:302] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d4adfed69e5416a813da1b4664030f7; no voters:
I20260808 21:06:48.293988 14934 leader_election.cc:288] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:48.294037 14939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:48.294240 14939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d4adfed69e5416a813da1b4664030f7, State: Running, Role: LEADER
I20260808 21:06:48.294284 14934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:48.294421 14939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } }
I20260808 21:06:48.294854 14942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d4adfed69e5416a813da1b4664030f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } } }
I20260808 21:06:48.294922 14942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:48.295154 14949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:48.295226 14941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d4adfed69e5416a813da1b4664030f7. Latest consensus state: current_term: 1 leader_uuid: "9d4adfed69e5416a813da1b4664030f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d4adfed69e5416a813da1b4664030f7" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 43657 } } }
I20260808 21:06:48.295334 14941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:48.295815 14949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:48.299391 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.293175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.4.177.126:43657)
0808 21:06:48.293448 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.293458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.293496 (+ 38us) server_negotiation.cc:407] Connection header received
0808 21:06:48.293621 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.293629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.293722 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.293880 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.294185 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.296192 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.297232 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.297813 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.297941 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.298642 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:48.298647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:48.298647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:48.298862 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:48.298903 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:48.298918 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:48.298992 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:48.299051 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 21:06:48.299132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
I20260808 21:06:48.299659 14949 catalog_manager.cc:1347] Generated new cluster ID: 9d9d8f21289c4f668597619011074cdd
I20260808 21:06:48.299736 14949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:48.344008 14949 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:48.344569 14949 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:48.351181 14949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d4adfed69e5416a813da1b4664030f7: Generated new TSK 0
I20260808 21:06:48.351406 14949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:48.360245 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--builtin_ntp_servers=127.4.177.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 21:06:48.458989 14963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:48.459259 14963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:48.459286 14963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:48.460470 14963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:48.460510 14963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:48.460536 14963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:48.461920 14963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:48.462211 14963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:48.462538 14963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:48.467460 14968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.467535 14969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.467497 14972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.467687 14963 server_base.cc:1034] running on GCE node
I20260808 21:06:48.467909 14963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:48.468163 14963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:48.469326 14963 hybrid_clock.cc:648] HybridClock initialized: now 1786223208469310 us; error 34 us; skew 500 ppm
I20260808 21:06:48.470566 14963 webserver.cc:460] Webserver started at http://127.4.177.65:32909/ using document root <none> and password file <none>
I20260808 21:06:48.470786 14963 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:48.470829 14963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:48.470923 14963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:48.472029 14963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "8bf7bd7d23a34d29917343793188f059"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "a8a998dfc178fbbc22f2ed4939feceec"
server_key_iv: "00e49eaaa5a8c21df04c78da350cf5df"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.472327 14963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "8bf7bd7d23a34d29917343793188f059"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "a8a998dfc178fbbc22f2ed4939feceec"
server_key_iv: "00e49eaaa5a8c21df04c78da350cf5df"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.473567 14963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:06:48.474261 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.474516 14963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:06:48.474603 14963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "8bf7bd7d23a34d29917343793188f059"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "a8a998dfc178fbbc22f2ed4939feceec"
server_key_iv: "00e49eaaa5a8c21df04c78da350cf5df"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.474659 14963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:48.483635 14963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:48.483863 14963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:48.483963 14963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:48.484117 14963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:48.484442 14963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:48.484477 14963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.484529 14963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:48.484556 14963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.490099 14963 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:41599
I20260808 21:06:48.490180 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:41599 every 8 connection(s)
I20260808 21:06:48.490513 14963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:48.495723 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.491294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:45115 (local address 127.4.177.126:43657)
0808 21:06:48.491366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.491369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.491619 (+ 250us) server_negotiation.cc:407] Connection header received
0808 21:06:48.492091 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.492092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.492104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.492161 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.492845 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.493337 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.494278 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.494478 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.494585 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.495443 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:48.495445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:48.495445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:48.495451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:48.495464 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:48.495469 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:48.495515 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:48.495595 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 21:06:48.495651 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:06:48.495927 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.491290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43657 (local address 127.4.177.65:45115)
0808 21:06:48.491505 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:48.491538 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:48.492009 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:48.492227 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:48.492230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:48.492238 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:48.492753 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.492762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.493415 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.493416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.494179 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.494181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.494604 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.494605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.494715 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.495249 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:48.495250 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:48.495276 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:48.495297 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:48.495526 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:48.495531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:48.495534 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:48.495585 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 21:06:48.495704 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260808 21:06:48.496502 15105 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43657
I20260808 21:06:48.496595 15105 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:48.496801 15105 heartbeater.cc:511] Master 127.4.177.126:43657 requested a full tablet report, sending...
I20260808 21:06:48.497129 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 14963
I20260808 21:06:48.497251 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:48.497387 14893 ts_manager.cc:194] Registered new tserver with Master: 8bf7bd7d23a34d29917343793188f059 (127.4.177.65:41599)
I20260808 21:06:48.497447 4805 external_mini_cluster.cc:1428] Setting key 8283b2f5eb52d19608d8c76313d4e4c6
I20260808 21:06:48.498198 14893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.65:45115
I20260808 21:06:48.498649 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--builtin_ntp_servers=127.4.177.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 21:06:48.580307 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:48.580503 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:48.580524 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:48.581666 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:48.581710 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:48.581735 15108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:48.583051 15108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:48.583351 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:48.583670 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:48.588640 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.588661 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.588683 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.588982 15108 server_base.cc:1034] running on GCE node
I20260808 21:06:48.589195 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:48.589498 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:48.590656 15108 hybrid_clock.cc:648] HybridClock initialized: now 1786223208590626 us; error 39 us; skew 500 ppm
I20260808 21:06:48.591966 15108 webserver.cc:460] Webserver started at http://127.4.177.66:44047/ using document root <none> and password file <none>
I20260808 21:06:48.592207 15108 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:48.592278 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:48.592414 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:48.593472 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "135fa3d06c9f43c5862af129421e9631"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "467af14bec3ee8381a554d39fc46b2d9"
server_key_iv: "c6530b916018ec870ef29e96ccb5e379"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.593842 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "135fa3d06c9f43c5862af129421e9631"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "467af14bec3ee8381a554d39fc46b2d9"
server_key_iv: "c6530b916018ec870ef29e96ccb5e379"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.595389 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:48.596254 15124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.596464 15108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:48.596570 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "135fa3d06c9f43c5862af129421e9631"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "467af14bec3ee8381a554d39fc46b2d9"
server_key_iv: "c6530b916018ec870ef29e96ccb5e379"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.596666 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:48.614591 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:48.614928 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:48.615093 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:48.615362 15108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:48.615726 15108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:48.615782 15108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.615825 15108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:48.615861 15108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.621762 15108 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38255
I20260808 21:06:48.621806 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38255 every 8 connection(s)
I20260808 21:06:48.622188 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:48.624004 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15108
I20260808 21:06:48.624081 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:48.624276 4805 external_mini_cluster.cc:1428] Setting key 6c50db61c614c212307f6713d66c98f3
I20260808 21:06:48.625325 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--builtin_ntp_servers=127.4.177.84:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 21:06:48.628327 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.622976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:53783 (local address 127.4.177.126:43657)
0808 21:06:48.623048 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.623050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.623289 (+ 239us) server_negotiation.cc:407] Connection header received
0808 21:06:48.623770 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.623771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.623795 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.623881 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.624694 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.625410 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.626888 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.627078 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.627193 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.628043 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:48.628044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:48.628044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:48.628049 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:48.628063 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:48.628067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:48.628112 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:48.628217 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 21:06:48.628247 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:06:48.628541 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.622955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43657 (local address 127.4.177.66:53783)
0808 21:06:48.623133 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:48.623197 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:48.623685 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:48.623944 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:48.623948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:48.623956 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:48.624619 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.624629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.625527 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.625528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.626799 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.626802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.627198 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.627199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.627336 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.627902 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:48.627902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:48.627930 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:48.627951 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:48.628122 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:48.628127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:48.628130 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:06:48.628216 (+ 86us) client_negotiation.cc:240] Negotiation successful
0808 21:06:48.628315 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260808 21:06:48.628996 15249 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43657
I20260808 21:06:48.629098 15249 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:48.629282 15249 heartbeater.cc:511] Master 127.4.177.126:43657 requested a full tablet report, sending...
I20260808 21:06:48.629657 14893 ts_manager.cc:194] Registered new tserver with Master: 135fa3d06c9f43c5862af129421e9631 (127.4.177.66:38255)
I20260808 21:06:48.630223 14893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.66:53783
W20260808 21:06:48.708745 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:48.708945 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:48.708966 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:48.710027 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:48.710059 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:48.710084 15252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:48.711410 15252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:48.711596 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:48.711840 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:48.716799 15261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.716776 15259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.716802 15258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.717072 15252 server_base.cc:1034] running on GCE node
I20260808 21:06:48.717300 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:48.717583 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:48.718739 15252 hybrid_clock.cc:648] HybridClock initialized: now 1786223208718715 us; error 37 us; skew 500 ppm
I20260808 21:06:48.720121 15252 webserver.cc:460] Webserver started at http://127.4.177.67:34037/ using document root <none> and password file <none>
I20260808 21:06:48.720373 15252 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:48.720444 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:48.720605 15252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:48.721721 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "ff7e4aeecb964f5891b89e05454549c6"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "f12a6394332d5ab7daa7742830c4895b"
server_key_iv: "2b267dee16e7065d1803516cec61949f"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.722105 15252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "ff7e4aeecb964f5891b89e05454549c6"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "f12a6394332d5ab7daa7742830c4895b"
server_key_iv: "2b267dee16e7065d1803516cec61949f"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.723474 15252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:48.724213 15267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.724412 15252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:06:48.724517 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "ff7e4aeecb964f5891b89e05454549c6"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "f12a6394332d5ab7daa7742830c4895b"
server_key_iv: "2b267dee16e7065d1803516cec61949f"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.724591 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:48.745888 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:48.746239 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:48.746402 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:48.746645 15252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:48.746990 15252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:48.747051 15252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.747084 15252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:48.747112 15252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.752875 15252 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:38915
I20260808 21:06:48.752928 15390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:38915 every 8 connection(s)
I20260808 21:06:48.753257 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:48.758436 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.754008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:54807 (local address 127.4.177.126:43657)
0808 21:06:48.754094 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.754096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.754284 (+ 188us) server_negotiation.cc:407] Connection header received
0808 21:06:48.754762 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.754763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.754773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.754826 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.755465 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.755949 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.756866 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.757042 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.757091 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.758175 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:48.758177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:48.758177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:48.758182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:48.758199 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:48.758203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:48.758256 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:48.758352 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 21:06:48.758376 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 21:06:48.758705 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.753952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:43657 (local address 127.4.177.67:54807)
0808 21:06:48.754164 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:48.754198 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:48.754680 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:48.754878 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:48.754881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:48.754889 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:48.755397 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.755401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.756027 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.756028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.756794 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:48.756796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:48.757270 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:48.757271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:48.757398 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:48.757931 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:48.757931 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:48.757961 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:48.757985 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:48.758274 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:48.758279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:48.758281 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:48.758342 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 21:06:48.758436 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260808 21:06:48.759277 15391 heartbeater.cc:347] Connected to a master server at 127.4.177.126:43657
I20260808 21:06:48.759395 15391 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:48.759600 15391 heartbeater.cc:511] Master 127.4.177.126:43657 requested a full tablet report, sending...
I20260808 21:06:48.760001 14893 ts_manager.cc:194] Registered new tserver with Master: ff7e4aeecb964f5891b89e05454549c6 (127.4.177.67:38915)
I20260808 21:06:48.760329 14893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.67:54807
I20260808 21:06:48.761445 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15252
I20260808 21:06:48.761559 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:48.761755 4805 external_mini_cluster.cc:1428] Setting key db0049be1907709df08d5e021aeea371
I20260808 21:06:48.762696 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 21:06:48.764688 14938 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:48.764244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.4.177.126:43657)
0808 21:06:48.764292 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.764294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.764401 (+ 107us) server_negotiation.cc:407] Connection header received
0808 21:06:48.764427 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.764427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.764437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.764497 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.764602 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54814: BlockingRecv error: recv got EOF from 127.0.0.1:54814 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:06:48.764818 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14963
I20260808 21:06:48.774376 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15108
I20260808 21:06:48.785457 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15252
I20260808 21:06:48.790998 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 14861
2026-08-08T21:06:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (662 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T21:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:48Z Disabled control of system clock
I20260808 21:06:48.819471 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34957
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:34957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 21:06:48.901343 15403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:48.901546 15403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:48.901576 15403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:48.902665 15403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:48.902701 15403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:48.902714 15403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:48.902727 15403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:48.902740 15403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:48.904125 15403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:34957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:48.904322 15403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:48.904551 15403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:48.909251 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:48.909323 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.909401 15403 server_base.cc:1034] running on GCE node
W20260808 21:06:48.909278 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:48.909677 15403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:48.909969 15403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:48.911115 15403 hybrid_clock.cc:648] HybridClock initialized: now 1786223208911101 us; error 27 us; skew 500 ppm
I20260808 21:06:48.912370 15403 webserver.cc:460] Webserver started at http://127.4.177.126:44619/ using document root <none> and password file <none>
I20260808 21:06:48.912576 15403 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:48.912618 15403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:48.912710 15403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:48.913789 15403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "1474d9e1ce1c4b15a385733ec8d07473"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "956050473add4e4ebfdd3b6dd6cb6c3e"
server_key_iv: "54b2989f201b8313c48149144ae2d2a8"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.914099 15403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "1474d9e1ce1c4b15a385733ec8d07473"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "956050473add4e4ebfdd3b6dd6cb6c3e"
server_key_iv: "54b2989f201b8313c48149144ae2d2a8"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.915393 15403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:06:48.916077 15417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.916249 15403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:48.916355 15403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "1474d9e1ce1c4b15a385733ec8d07473"
format_stamp: "Formatted at 2026-08-08 21:06:48 on dist-test-slave-chsk"
server_key: "956050473add4e4ebfdd3b6dd6cb6c3e"
server_key_iv: "54b2989f201b8313c48149144ae2d2a8"
server_key_version: "encryptionkey@0"
I20260808 21:06:48.916431 15403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:48.919863 15403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:48.920145 15403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:48.920290 15403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:48.924472 15403 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:34957
I20260808 21:06:48.924504 15473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:34957 every 8 connection(s)
I20260808 21:06:48.924836 15403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:48.925467 15474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:48.928227 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473: Bootstrap starting.
I20260808 21:06:48.928790 15474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:48.929029 15474 log.cc:824] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:48.929735 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473: No bootstrap required, opened a new log
I20260808 21:06:48.931360 15474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } }
I20260808 21:06:48.931492 15474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:48.931514 15474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1474d9e1ce1c4b15a385733ec8d07473, State: Initialized, Role: FOLLOWER
I20260808 21:06:48.931654 15474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } }
I20260808 21:06:48.931733 15474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:48.931756 15474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:48.931784 15474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:48.932266 15474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } }
I20260808 21:06:48.932351 15474 leader_election.cc:302] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1474d9e1ce1c4b15a385733ec8d07473; no voters:
I20260808 21:06:48.932466 15474 leader_election.cc:288] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:48.932535 15477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:48.932735 15474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:48.932750 15477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [term 1 LEADER]: Becoming Leader. State: Replica: 1474d9e1ce1c4b15a385733ec8d07473, State: Running, Role: LEADER
I20260808 21:06:48.932883 15477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } }
I20260808 21:06:48.933343 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1474d9e1ce1c4b15a385733ec8d07473. Latest consensus state: current_term: 1 leader_uuid: "1474d9e1ce1c4b15a385733ec8d07473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } } }
I20260808 21:06:48.933434 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:48.933689 15478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1474d9e1ce1c4b15a385733ec8d07473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1474d9e1ce1c4b15a385733ec8d07473" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34957 } } }
I20260808 21:06:48.933825 15478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:48.934031 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15403
I20260808 21:06:48.934137 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:48.934365 4805 external_mini_cluster.cc:1428] Setting key bf4a7a6d10f7646495f71147fce14614
I20260808 21:06:48.934548 15489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:48.935065 15489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:48.936758 15489 catalog_manager.cc:1347] Generated new cluster ID: 7a75f2ee1d364601bddbd8c0a0a08bec
I20260808 21:06:48.936831 15489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:06:48.940349 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:48.934881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.4.177.126:34957)
0808 21:06:48.935064 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:48.935080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:48.935368 (+ 288us) server_negotiation.cc:407] Connection header received
0808 21:06:48.935692 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:48.935698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:48.935754 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:48.935885 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:48.939923 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:48.939926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:48.939927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:48.939931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:48.939986 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:48.939997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:48.940066 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:48.940148 (+ 82us) server_negotiation.cc:299] Negotiation successful
0808 21:06:48.940205 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260808 21:06:48.953037 15489 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:48.953235 15489 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:48.958839 15489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1474d9e1ce1c4b15a385733ec8d07473: Generated new TSK 0
I20260808 21:06:48.959040 15489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:06:48.964877 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--builtin_ntp_servers=127.4.177.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 21:06:49.061414 15501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:49.061617 15501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:49.061640 15501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:49.062821 15501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:49.062861 15501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:49.062888 15501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:49.064275 15501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:49.064507 15501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:49.064814 15501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:49.070611 15507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:49.070843 15501 server_base.cc:1034] running on GCE node
W20260808 21:06:49.070611 15506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:49.070669 15509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:49.071108 15501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:49.071420 15501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:49.072587 15501 hybrid_clock.cc:648] HybridClock initialized: now 1786223209072574 us; error 29 us; skew 500 ppm
I20260808 21:06:49.073856 15501 webserver.cc:460] Webserver started at http://127.4.177.65:39589/ using document root <none> and password file <none>
I20260808 21:06:49.074081 15501 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:49.074160 15501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:49.074295 15501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:49.075491 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "c08f8b3434bc4158ac0b698fac6cd847"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "b78dc8836091bd828c26be3763ab97b1"
server_key_iv: "18e6cd3aceb046b2ea920a3ed97b310e"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.075862 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "c08f8b3434bc4158ac0b698fac6cd847"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "b78dc8836091bd828c26be3763ab97b1"
server_key_iv: "18e6cd3aceb046b2ea920a3ed97b310e"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.077208 15501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:49.077991 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.078184 15501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:49.078284 15501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "c08f8b3434bc4158ac0b698fac6cd847"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "b78dc8836091bd828c26be3763ab97b1"
server_key_iv: "18e6cd3aceb046b2ea920a3ed97b310e"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.078369 15501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:49.081818 15501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:49.082100 15501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:49.082253 15501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:49.082491 15501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:49.082801 15501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:49.082856 15501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.082895 15501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:49.082922 15501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.089502 15501 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:45369
I20260808 21:06:49.089553 15637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:45369 every 8 connection(s)
I20260808 21:06:49.089883 15501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:06:49.091169 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15501
I20260808 21:06:49.091279 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:49.091483 4805 external_mini_cluster.cc:1428] Setting key 9da7e2a94abb97a8a60c941d4981bd9b
I20260808 21:06:49.092514 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--builtin_ntp_servers=127.4.177.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 21:06:49.093345 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.090655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:42655 (local address 127.4.177.126:34957)
0808 21:06:49.090719 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:49.090721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:49.091662 (+ 941us) server_negotiation.cc:407] Connection header received
0808 21:06:49.091694 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:49.091695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:49.091704 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:49.091756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:49.093049 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:49.093051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:49.093052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:49.093057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:49.093075 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:49.093080 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:49.093148 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:49.093226 (+ 78us) server_negotiation.cc:299] Negotiation successful
0808 21:06:49.093243 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:06:49.093439 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.090648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34957 (local address 127.4.177.65:42655)
0808 21:06:49.090841 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:49.090875 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:49.091567 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:49.091837 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:49.091841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:49.091850 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:49.092905 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:49.092906 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:49.092933 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:49.092960 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:49.093154 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:49.093157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:49.093159 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:49.093211 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 21:06:49.093258 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260808 21:06:49.093952 15638 heartbeater.cc:347] Connected to a master server at 127.4.177.126:34957
I20260808 21:06:49.094056 15638 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:49.094202 15638 heartbeater.cc:511] Master 127.4.177.126:34957 requested a full tablet report, sending...
I20260808 21:06:49.094790 15436 ts_manager.cc:194] Registered new tserver with Master: c08f8b3434bc4158ac0b698fac6cd847 (127.4.177.65:45369)
W20260808 21:06:49.174976 15641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:49.175226 15641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:49.175266 15641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:49.176455 15641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:49.176501 15641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:49.176528 15641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:49.177888 15641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:49.178138 15641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:49.178455 15641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:49.183808 15647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:49.183888 15641 server_base.cc:1034] running on GCE node
W20260808 21:06:49.183779 15650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:49.184000 15648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:49.184222 15641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:49.184538 15641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:49.185699 15641 hybrid_clock.cc:648] HybridClock initialized: now 1786223209185693 us; error 34 us; skew 500 ppm
I20260808 21:06:49.186992 15641 webserver.cc:460] Webserver started at http://127.4.177.66:41123/ using document root <none> and password file <none>
I20260808 21:06:49.187302 15641 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:49.187371 15641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:49.187511 15641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:49.188553 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "1203b002238f41abbfe471ed7233743e"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "3415f471a694e3272a167a3c1712a6fe"
server_key_iv: "ab281f8327a2eec66857f24acf1622a7"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.188903 15641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "1203b002238f41abbfe471ed7233743e"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "3415f471a694e3272a167a3c1712a6fe"
server_key_iv: "ab281f8327a2eec66857f24acf1622a7"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.190315 15641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:49.191063 15657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.191341 15641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:49.191413 15641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "1203b002238f41abbfe471ed7233743e"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "3415f471a694e3272a167a3c1712a6fe"
server_key_iv: "ab281f8327a2eec66857f24acf1622a7"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.191500 15641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:49.194746 15641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:49.194994 15641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:49.195123 15641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:49.195358 15641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:49.195662 15641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:49.195694 15641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.195745 15641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:49.195775 15641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.201642 15641 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:35467
I20260808 21:06:49.201691 15780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:35467 every 8 connection(s)
I20260808 21:06:49.202021 15641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:49.204907 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.202762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:54719 (local address 127.4.177.126:34957)
0808 21:06:49.202829 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:49.202831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:49.203062 (+ 231us) server_negotiation.cc:407] Connection header received
0808 21:06:49.203574 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:49.203575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:49.203580 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:49.203647 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:49.204706 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:49.204707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:49.204707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:49.204712 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:49.204727 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:49.204731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:49.204783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:49.204829 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 21:06:49.204841 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:06:49.205173 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.202736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34957 (local address 127.4.177.66:54719)
0808 21:06:49.202923 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:49.202964 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:49.203483 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:49.203699 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:49.203702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:49.203711 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:49.204563 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:49.204563 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:49.204590 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:49.204617 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:49.204779 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:49.204782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:49.204784 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:49.204835 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 21:06:49.204893 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260808 21:06:49.205713 15781 heartbeater.cc:347] Connected to a master server at 127.4.177.126:34957
I20260808 21:06:49.205793 15781 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:49.205964 15781 heartbeater.cc:511] Master 127.4.177.126:34957 requested a full tablet report, sending...
I20260808 21:06:49.206331 15436 ts_manager.cc:194] Registered new tserver with Master: 1203b002238f41abbfe471ed7233743e (127.4.177.66:35467)
I20260808 21:06:49.208405 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15641
I20260808 21:06:49.208534 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:49.208747 4805 external_mini_cluster.cc:1428] Setting key 1e3fde5b8cbec90d003c50163d388cd4
I20260808 21:06:49.209674 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--builtin_ntp_servers=127.4.177.84:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 21:06:49.292282 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:49.292484 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:49.292506 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:49.293690 15785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:49.293730 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:49.293756 15785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:49.295288 15785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:34957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:49.295467 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:49.295703 15785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:49.300393 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:49.300446 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:49.300369 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:49.300575 15785 server_base.cc:1034] running on GCE node
I20260808 21:06:49.300789 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:49.300983 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:49.302130 15785 hybrid_clock.cc:648] HybridClock initialized: now 1786223209302119 us; error 27 us; skew 500 ppm
I20260808 21:06:49.303483 15785 webserver.cc:460] Webserver started at http://127.4.177.67:42553/ using document root <none> and password file <none>
I20260808 21:06:49.303727 15785 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:49.303776 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:49.303946 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:49.305016 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "b777aef258e14b88a692ccf1853a9ae9"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "46adccbbbc2f183e53ff532e1cf0a960"
server_key_iv: "e70f844e49808e61db0ed2acde44fd83"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.305392 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "b777aef258e14b88a692ccf1853a9ae9"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "46adccbbbc2f183e53ff532e1cf0a960"
server_key_iv: "e70f844e49808e61db0ed2acde44fd83"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.306728 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:06:49.307464 15800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.307682 15785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.307789 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "b777aef258e14b88a692ccf1853a9ae9"
format_stamp: "Formatted at 2026-08-08 21:06:49 on dist-test-slave-chsk"
server_key: "46adccbbbc2f183e53ff532e1cf0a960"
server_key_iv: "e70f844e49808e61db0ed2acde44fd83"
server_key_version: "encryptionkey@0"
I20260808 21:06:49.307859 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:49.311190 15785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:49.311460 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:49.311600 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:49.311834 15785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:49.312134 15785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:49.312170 15785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.312222 15785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:49.312250 15785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:49.317723 15785 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:33995
I20260808 21:06:49.317795 15925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:33995 every 8 connection(s)
I20260808 21:06:49.318086 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:49.321008 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.318833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33321 (local address 127.4.177.126:34957)
0808 21:06:49.318892 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:49.318895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:49.319097 (+ 202us) server_negotiation.cc:407] Connection header received
0808 21:06:49.319597 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:49.319598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:49.319604 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:49.319656 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:49.320835 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:49.320836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:49.320836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:06:49.320839 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:49.320850 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:06:49.320853 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:49.320897 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:49.320945 (+ 48us) server_negotiation.cc:299] Negotiation successful
0808 21:06:49.320957 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:06:49.321219 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:49.318792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34957 (local address 127.4.177.67:33321)
0808 21:06:49.318993 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:49.319029 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:49.319509 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:49.319722 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:49.319725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:49.319732 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:49.320710 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:06:49.320710 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:49.320737 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:06:49.320763 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:49.320894 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:49.320896 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:49.320898 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:06:49.320946 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 21:06:49.320989 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260808 21:06:49.321713 15926 heartbeater.cc:347] Connected to a master server at 127.4.177.126:34957
I20260808 21:06:49.321786 15926 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:49.321947 15926 heartbeater.cc:511] Master 127.4.177.126:34957 requested a full tablet report, sending...
I20260808 21:06:49.322317 15436 ts_manager.cc:194] Registered new tserver with Master: b777aef258e14b88a692ccf1853a9ae9 (127.4.177.67:33995)
I20260808 21:06:49.325219 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15785
I20260808 21:06:49.325323 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:49.325516 4805 external_mini_cluster.cc:1428] Setting key 6c87e6919605321479d5790436da834a
I20260808 21:06:49.326359 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 21:06:49.328193 15499 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:06:49.327668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.4.177.126:34957)
0808 21:06:49.327723 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:49.327725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:49.327861 (+ 136us) server_negotiation.cc:407] Connection header received
0808 21:06:49.327940 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:49.327941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:49.327945 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:49.327990 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:49.328126 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52222: BlockingRecv error: recv got EOF from 127.0.0.1:52222 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260808 21:06:49.328243 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15501
I20260808 21:06:49.335681 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15641
I20260808 21:06:49.343206 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15785
I20260808 21:06:49.350970 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15403
2026-08-08T21:06:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:49 dist-test-slave-chsk krb5kdc[15938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:49 dist-test-slave-chsk krb5kdc[15938](info): set up 2 sockets
Aug 08 21:06:49 dist-test-slave-chsk krb5kdc[15938](info): commencing operation
krb5kdc: starting...
W20260808 21:06:50.477108 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:50 dist-test-slave-chsk krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:06:50.604168 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40989
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:50.687182 15953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:50.687382 15953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:50.687404 15953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:50.688476 15953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:06:50.688509 15953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:50.688524 15953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:50.688536 15953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:06:50.688548 15953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:06:50.689940 15953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:50.690140 15953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:50.690315 15953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:50.695087 15960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:50.695084 15953 server_base.cc:1034] running on GCE node
W20260808 21:06:50.695230 15959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:50.695393 15962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:50.695645 15953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:50.695891 15953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:50.697036 15953 hybrid_clock.cc:648] HybridClock initialized: now 1786223210697011 us; error 48 us; skew 500 ppm
Aug 08 21:06:50 dist-test-slave-chsk krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223210, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:50.699685 15953 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:06:50.700145 15953 webserver.cc:460] Webserver started at http://127.4.177.126:46585/ using document root <none> and password file <none>
I20260808 21:06:50.700346 15953 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:50.700388 15953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:50.700477 15953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:50.701471 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "36351c4145e8448f93a585a35af25cb1"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "7319e693656fdeb6a3654020a8e98bdf"
server_key_iv: "710eb352ddd8c8906d36c82309fa8411"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.701768 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "36351c4145e8448f93a585a35af25cb1"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "7319e693656fdeb6a3654020a8e98bdf"
server_key_iv: "710eb352ddd8c8906d36c82309fa8411"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.703002 15953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:50.703744 15970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:50.703902 15953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:50.703987 15953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "36351c4145e8448f93a585a35af25cb1"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "7319e693656fdeb6a3654020a8e98bdf"
server_key_iv: "710eb352ddd8c8906d36c82309fa8411"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.704042 15953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:50.715595 15953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:50.717352 15953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:50.717461 15953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:50.721410 15953 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:40989
I20260808 21:06:50.721472 16029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:40989 every 8 connection(s)
I20260808 21:06:50.721787 15953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:06:50.722769 16030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:50.725036 16030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: Bootstrap starting.
I20260808 21:06:50.725625 16030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:50.725925 16030 log.cc:824] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:50.726640 16030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: No bootstrap required, opened a new log
I20260808 21:06:50.728425 16030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } }
I20260808 21:06:50.728581 16030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:50.728660 16030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36351c4145e8448f93a585a35af25cb1, State: Initialized, Role: FOLLOWER
I20260808 21:06:50.728833 16030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } }
I20260808 21:06:50.728935 16030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:06:50.728978 16030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:06:50.729027 16030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:50.729723 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 15953
I20260808 21:06:50.729802 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:06:50.729849 16030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } }
I20260808 21:06:50.729974 16030 leader_election.cc:302] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36351c4145e8448f93a585a35af25cb1; no voters:
I20260808 21:06:50.730026 4805 external_mini_cluster.cc:1428] Setting key 5933ccb94f45f49c894f6a0a82c3a1f5
I20260808 21:06:50.730152 16030 leader_election.cc:288] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:06:50.730186 16033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:50.730309 16033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 36351c4145e8448f93a585a35af25cb1, State: Running, Role: LEADER
I20260808 21:06:50.730386 16033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } }
I20260808 21:06:50.730674 16030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:06:50.730697 16033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36351c4145e8448f93a585a35af25cb1. Latest consensus state: current_term: 1 leader_uuid: "36351c4145e8448f93a585a35af25cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } } }
I20260808 21:06:50.730713 16036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36351c4145e8448f93a585a35af25cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36351c4145e8448f93a585a35af25cb1" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40989 } } }
I20260808 21:06:50.730767 16033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:50.730916 16036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1 [sys.catalog]: This master's current role is: LEADER
I20260808 21:06:50.731114 16040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:06:50.731716 16040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:06:50.733409 16040 catalog_manager.cc:1347] Generated new cluster ID: 5f56e2aab89f4aafadac64e9c5ff4f3c
I20260808 21:06:50.733462 16040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:06:50 dist-test-slave-chsk krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:50.741291 16040 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:06:50.741963 16040 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:06:50.742874 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:50.730513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.4.177.126:40989)
0808 21:06:50.730908 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:50.730918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:50.731105 (+ 187us) server_negotiation.cc:407] Connection header received
0808 21:06:50.731362 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:50.731369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:50.731427 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:50.731543 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:50.732605 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.733967 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.734987 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.735645 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.735769 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:50.738556 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:50.738560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:50.738560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:50.738572 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:50.741212 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:50.741605 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:50.741609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:50.741609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:50.741724 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:50.741949 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:50.741950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:50.741951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:50.742239 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:50.742388 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:50.742504 (+ 116us) server_negotiation.cc:299] Negotiation successful
0808 21:06:50.742642 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20260808 21:06:50.747220 16040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: Generated new TSK 0
I20260808 21:06:50.747399 16040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:06:50.773589 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:50.856449 16057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:50.856652 16057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:50.856674 16057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:50.857811 16057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:50.857853 16057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:50.857880 16057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:06:50.859431 16057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:50.859669 16057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:50.859911 16057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:50.864672 16063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:50.864684 16067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:50.864707 16064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:50.865188 16057 server_base.cc:1034] running on GCE node
I20260808 21:06:50.865356 16057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:50.865540 16057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:50.866705 16057 hybrid_clock.cc:648] HybridClock initialized: now 1786223210866691 us; error 27 us; skew 500 ppm
Aug 08 21:06:50 dist-test-slave-chsk krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223210, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:50.869292 16057 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:06:50.869769 16057 webserver.cc:460] Webserver started at http://127.4.177.65:37495/ using document root <none> and password file <none>
I20260808 21:06:50.870002 16057 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:50.870055 16057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:50.870342 16057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:50.871539 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "4f43f06c83e4479f930c371892a1c50a"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "038a45f3758134cd761b3b77ed9d1c65"
server_key_iv: "b42c17237b42378a5c877ad40a1cf38f"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.871964 16057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "4f43f06c83e4479f930c371892a1c50a"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "038a45f3758134cd761b3b77ed9d1c65"
server_key_iv: "b42c17237b42378a5c877ad40a1cf38f"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.873293 16057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:06:50.874009 16074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:50.874241 16057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:50.874343 16057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "4f43f06c83e4479f930c371892a1c50a"
format_stamp: "Formatted at 2026-08-08 21:06:50 on dist-test-slave-chsk"
server_key: "038a45f3758134cd761b3b77ed9d1c65"
server_key_iv: "b42c17237b42378a5c877ad40a1cf38f"
server_key_version: "encryptionkey@0"
I20260808 21:06:50.874420 16057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:50.890378 16057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:50.892584 16057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:50.892760 16057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:50.893013 16057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:50.893340 16057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:50.893399 16057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:50.893446 16057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:50.893471 16057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:50.899552 16057 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:37649
I20260808 21:06:50.899606 16198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:37649 every 8 connection(s)
I20260808 21:06:50.899930 16057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:06:50 dist-test-slave-chsk krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223210, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:50.908900 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:50.900618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:60459 (local address 127.4.177.126:40989)
0808 21:06:50.900697 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:50.900699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:50.900931 (+ 232us) server_negotiation.cc:407] Connection header received
0808 21:06:50.901414 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:50.901415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:50.901424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:50.901483 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:50.902419 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.902936 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.903950 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.904136 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.904200 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:50.906373 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:50.906375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:50.906375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:50.906381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:50.907707 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:50.908065 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:50.908066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:50.908067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:50.908127 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:50.908349 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:50.908350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:50.908350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:50.908456 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:50.908508 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:50.908737 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 21:06:50.908810 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 21:06:50.909070 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:50.900619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40989 (local address 127.4.177.65:60459)
0808 21:06:50.900795 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:50.900837 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:50.901333 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:50.901537 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:50.901540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:50.902135 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:50.902325 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:50.902330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.903018 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.903019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:50.903875 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:50.903878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:50.904220 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:50.904221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:50.904319 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:50.904880 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:50.904888 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:50.906254 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:50.907779 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:50.907785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:50.907787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:50.907999 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:50.908194 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:50.908194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:50.908194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:50.908293 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:50.908510 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:50.908514 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:50.908631 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 21:06:50.908710 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:06:50.908832 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260808 21:06:50.909070 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 16057
I20260808 21:06:50.909165 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:06:50.909375 4805 external_mini_cluster.cc:1428] Setting key 29a06fd95fab1ee75c31115dc7b7364f
I20260808 21:06:50.909605 16199 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40989
I20260808 21:06:50.909826 16199 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:50.910017 16199 heartbeater.cc:511] Master 127.4.177.126:40989 requested a full tablet report, sending...
I20260808 21:06:50.910597 15989 ts_manager.cc:194] Registered new tserver with Master: 4f43f06c83e4479f930c371892a1c50a (127.4.177.65:37649)
I20260808 21:06:50.911528 15989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:60459
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:06:50.935381 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:51.017742 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:51.017989 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:51.018029 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:51.019209 16204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:51.019276 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:51.019320 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:06:51.020826 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:51.021106 16204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:51.021349 16204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:06:51.026592 16204 server_base.cc:1034] running on GCE node
W20260808 21:06:51.026510 16209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:51.026527 16212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:51.026607 16210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:51.027011 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:51.027318 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:51.028483 16204 hybrid_clock.cc:648] HybridClock initialized: now 1786223211028467 us; error 46 us; skew 500 ppm
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:51.031427 16204 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:06:51.032006 16204 webserver.cc:460] Webserver started at http://127.4.177.66:39361/ using document root <none> and password file <none>
I20260808 21:06:51.032270 16204 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:51.032341 16204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:51.032480 16204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:51.033533 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "4b4260a5d1454c75ba2ef2d999b55acc"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "ebd6c5781d1efd1f9a6d73112f2c14f4"
server_key_iv: "421525d13bc3bc89c33b346d083ee754"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.033905 16204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "4b4260a5d1454c75ba2ef2d999b55acc"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "ebd6c5781d1efd1f9a6d73112f2c14f4"
server_key_iv: "421525d13bc3bc89c33b346d083ee754"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.035336 16204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 21:06:51.036307 16219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.036502 16204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:06:51.036612 16204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "4b4260a5d1454c75ba2ef2d999b55acc"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "ebd6c5781d1efd1f9a6d73112f2c14f4"
server_key_iv: "421525d13bc3bc89c33b346d083ee754"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.036698 16204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:51.057576 16204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:51.060040 16204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:51.060212 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:51.060513 16204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:51.060844 16204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:51.060899 16204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.060935 16204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:51.060963 16204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.067251 16204 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:36393
I20260808 21:06:51.067308 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:36393 every 8 connection(s)
I20260808 21:06:51.067653 16204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:06:51.070729 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 16204
I20260808 21:06:51.070833 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:06:51.071033 4805 external_mini_cluster.cc:1428] Setting key c1fcef523734d735b047593b05063ede
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:51.078001 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.068394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:35611 (local address 127.4.177.126:40989)
0808 21:06:51.068478 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.068481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.068647 (+ 166us) server_negotiation.cc:407] Connection header received
0808 21:06:51.069145 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.069146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.069158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.069218 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:51.070158 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.070755 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.072156 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.072441 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.072519 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.074977 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:51.074978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:51.074978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:51.074985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:51.076441 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.076963 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.076964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.076965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.077015 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.077328 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.077329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.077330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.077449 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:51.077522 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.077819 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.077905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:06:51.078213 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.068337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40989 (local address 127.4.177.66:35611)
0808 21:06:51.068521 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.068559 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.069034 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.069278 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.069281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.069867 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:51.070084 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.070090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.070893 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.070895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.072072 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.072075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.072530 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.072531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.072635 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.073202 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:51.073210 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:51.074884 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:51.076537 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:51.076546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:51.076548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:51.076885 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:51.077101 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:51.077102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:51.077103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:51.077245 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:51.077525 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:51.077534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:51.077678 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.077784 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.077953 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260808 21:06:51.078693 16344 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40989
I20260808 21:06:51.078790 16344 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:51.078990 16344 heartbeater.cc:511] Master 127.4.177.126:40989 requested a full tablet report, sending...
I20260808 21:06:51.079422 15989 ts_manager.cc:194] Registered new tserver with Master: 4b4260a5d1454c75ba2ef2d999b55acc (127.4.177.66:36393)
I20260808 21:06:51.079965 15989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:35611
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:06:51.097987 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:06:51.181713 16349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:06:51.181931 16349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:06:51.181957 16349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:06:51.183255 16349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:06:51.183303 16349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:06:51.183332 16349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:06:51.184929 16349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:06:51.185189 16349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:06:51.185461 16349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:06:51.190771 16357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:51.190804 16349 server_base.cc:1034] running on GCE node
W20260808 21:06:51.190771 16355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:06:51.190810 16354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:06:51.191275 16349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:06:51.191548 16349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:06:51.192734 16349 hybrid_clock.cc:648] HybridClock initialized: now 1786223211192721 us; error 41 us; skew 500 ppm
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[15938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:06:51.195510 16349 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:06:51.195923 16349 webserver.cc:460] Webserver started at http://127.4.177.67:33503/ using document root <none> and password file <none>
I20260808 21:06:51.196138 16349 fs_manager.cc:359] Metadata directory not provided
I20260808 21:06:51.196192 16349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:06:51.196285 16349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:06:51.197304 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "4a7417011d154f81a783509b665e5dd6"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "14794e1aaa1615f60e644f34bea034e3"
server_key_iv: "416775041f924543f8c538399383c98a"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.197623 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "4a7417011d154f81a783509b665e5dd6"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "14794e1aaa1615f60e644f34bea034e3"
server_key_iv: "416775041f924543f8c538399383c98a"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.198906 16349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 21:06:51.199692 16364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.199867 16349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.199958 16349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "4a7417011d154f81a783509b665e5dd6"
format_stamp: "Formatted at 2026-08-08 21:06:51 on dist-test-slave-chsk"
server_key: "14794e1aaa1615f60e644f34bea034e3"
server_key_iv: "416775041f924543f8c538399383c98a"
server_key_version: "encryptionkey@0"
I20260808 21:06:51.200013 16349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:06:51.208629 16349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:06:51.210994 16349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:06:51.211124 16349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:06:51.211402 16349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:06:51.211665 16349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:06:51.211696 16349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.211719 16349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:06:51.211731 16349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:06:51.217489 16349 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:33535
I20260808 21:06:51.217576 16487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:33535 every 8 connection(s)
I20260808 21:06:51.217911 16349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:06:51.223874 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 16349
I20260808 21:06:51.224004 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:06:51.224237 4805 external_mini_cluster.cc:1428] Setting key 3e536430803c3fdc244e651e948a1ec9
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[15938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223211, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:06:51.227955 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.218747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:54153 (local address 127.4.177.126:40989)
0808 21:06:51.218820 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.218822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.219052 (+ 230us) server_negotiation.cc:407] Connection header received
0808 21:06:51.219688 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.219690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.219704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.219756 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:51.220756 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.221273 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.222311 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.222514 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.222589 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.225221 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:51.225222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:51.225223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:51.225232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:51.226587 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.227014 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.227015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.227015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.227078 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.227395 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.227396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.227396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.227514 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:51.227571 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.227819 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.227888 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:06:51.228121 16490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.218713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40989 (local address 127.4.177.67:54153)
0808 21:06:51.218922 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.218959 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.219555 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.219826 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.219830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.220433 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:06:51.220648 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.220653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.221383 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.221384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.222214 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.222216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.222620 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.222621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.222788 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.223457 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:06:51.223468 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:06:51.225123 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:06:51.226657 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:51.226663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:51.226665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:51.226895 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:51.227219 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:06:51.227220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:06:51.227220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:06:51.227333 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:06:51.227573 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:06:51.227578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:06:51.227699 (+ 121us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.227777 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.227904 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260808 21:06:51.228601 16488 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40989
I20260808 21:06:51.228714 16488 heartbeater.cc:464] Registering TS with master...
I20260808 21:06:51.228911 16488 heartbeater.cc:511] Master 127.4.177.126:40989 requested a full tablet report, sending...
I20260808 21:06:51.229314 15989 ts_manager.cc:194] Registered new tserver with Master: 4a7417011d154f81a783509b665e5dd6 (127.4.177.67:33535)
I20260808 21:06:51.229691 15989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:54153
I20260808 21:06:51.236271 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:06:51.244377 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.237699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.4.177.126:40989)
0808 21:06:51.237781 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.237784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.237857 (+ 73us) server_negotiation.cc:407] Connection header received
0808 21:06:51.237935 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.237936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.237946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.238017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:06:51.238647 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.239208 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.240033 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.240233 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.240309 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.241626 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:06:51.241628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:06:51.241628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:06:51.241635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:06:51.243118 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.243502 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.243503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.243503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.243565 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:06:51.243781 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:06:51.243782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:06:51.243782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:06:51.243941 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:06:51.244047 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.244177 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.244262 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 21:06:51.245980 15989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:06:51.247825 15989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:06:51.254632 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.4.177.66:36393)
0808 21:06:51.250470 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.250476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.250728 (+ 252us) server_negotiation.cc:407] Connection header received
0808 21:06:51.250774 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.250777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.250792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.250893 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:51.251093 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.251986 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.253686 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254173 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254271 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.254331 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.254496 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.254534 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260808 21:06:51.254630 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:36393 (local address 127.0.0.1:40120)
0808 21:06:51.250512 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.250553 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.250634 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.250891 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.250892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.250898 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:51.250997 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.250999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.252103 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.252104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.253605 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.253607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254277 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.254361 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.254374 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.254477 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.254526 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260808 21:06:51.254632 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:33535 (local address 127.0.0.1:41638)
0808 21:06:51.251003 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.251014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.251056 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.251419 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.251420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.251423 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:51.251459 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.251460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.252597 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.252598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.253345 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.253347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254300 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.254370 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.254375 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.254422 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.254491 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260808 21:06:51.254665 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.4.177.65:37649)
0808 21:06:51.250695 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.250700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.250711 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:06:51.250755 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.250757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.250772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.250878 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:51.251100 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.252249 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.253512 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254200 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254266 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.254330 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.254537 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.254584 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260808 21:06:51.255368 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.4.177.67:33535)
0808 21:06:51.251205 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.251210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.251219 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:06:51.251268 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.251270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.251285 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.251393 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:51.251536 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.252503 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.253435 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254179 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254660 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.255187 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.255212 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.255284 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":84,"threads_started":1}
I20260808 21:06:51.255535 16271 tablet_service.cc:1467] Processing CreateTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 (DEFAULT_TABLE table=test-table [id=64a0d4454a0a4c62a13dc4f0f1f2af09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:51.255702 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.250180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37649 (local address 127.0.0.1:49904)
0808 21:06:51.250610 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.250621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.250680 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.250888 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.250891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.250894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:51.251005 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.251008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.252363 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.252364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.253438 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.253440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.254358 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.254359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.254413 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.254417 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.254462 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.254498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260808 21:06:51.255934 16271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca336a1fa11a47609e1d05babd5f0b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:51.256014 16416 tablet_service.cc:1467] Processing CreateTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 (DEFAULT_TABLE table=test-table [id=64a0d4454a0a4c62a13dc4f0f1f2af09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:51.256330 16416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca336a1fa11a47609e1d05babd5f0b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:51.256558 16125 tablet_service.cc:1467] Processing CreateTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 (DEFAULT_TABLE table=test-table [id=64a0d4454a0a4c62a13dc4f0f1f2af09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:06:51.256868 16125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca336a1fa11a47609e1d05babd5f0b56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:06:51.259692 16509 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: Bootstrap starting.
I20260808 21:06:51.260351 16509 tablet_bootstrap.cc:654] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:51.260886 16509 log.cc:824] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:51.261234 16508 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Bootstrap starting.
I20260808 21:06:51.261869 16509 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: No bootstrap required, opened a new log
I20260808 21:06:51.261916 16508 tablet_bootstrap.cc:654] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:51.261952 16509 ts_tablet_manager.cc:1397] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:51.262481 16508 log.cc:824] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:51.262843 16506 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Bootstrap starting.
I20260808 21:06:51.263326 16508 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: No bootstrap required, opened a new log
I20260808 21:06:51.263408 16508 ts_tablet_manager.cc:1397] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:06:51.263482 16506 tablet_bootstrap.cc:654] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Neither blocks nor log segments found. Creating new log.
I20260808 21:06:51.264075 16506 log.cc:824] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Log is configured to *not* fsync() on all Append() calls
I20260808 21:06:51.264364 16509 raft_consensus.cc:348] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.264530 16509 raft_consensus.cc:374] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:51.264575 16509 raft_consensus.cc:729] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f43f06c83e4479f930c371892a1c50a, State: Initialized, Role: FOLLOWER
I20260808 21:06:51.264684 16509 consensus_queue.cc:260] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.264846 16506 tablet_bootstrap.cc:492] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: No bootstrap required, opened a new log
I20260808 21:06:51.264917 16506 ts_tablet_manager.cc:1397] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:06:51.265004 16509 ts_tablet_manager.cc:1428] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:06:51.265074 16199 heartbeater.cc:503] Master 127.4.177.126:40989 was elected leader, sending a full tablet report...
I20260808 21:06:51.265300 16508 raft_consensus.cc:348] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.265475 16508 raft_consensus.cc:374] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:51.265514 16508 raft_consensus.cc:729] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4260a5d1454c75ba2ef2d999b55acc, State: Initialized, Role: FOLLOWER
I20260808 21:06:51.265636 16508 consensus_queue.cc:260] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.265915 16508 ts_tablet_manager.cc:1428] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:06:51.266052 16344 heartbeater.cc:503] Master 127.4.177.126:40989 was elected leader, sending a full tablet report...
I20260808 21:06:51.267169 16506 raft_consensus.cc:348] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.267306 16506 raft_consensus.cc:374] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:06:51.267328 16506 raft_consensus.cc:729] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7417011d154f81a783509b665e5dd6, State: Initialized, Role: FOLLOWER
I20260808 21:06:51.267397 16506 consensus_queue.cc:260] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.267596 16506 ts_tablet_manager.cc:1428] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:06:51.267644 16488 heartbeater.cc:503] Master 127.4.177.126:40989 was elected leader, sending a full tablet report...
I20260808 21:06:51.309046 16514 raft_consensus.cc:482] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:06:51.309216 16514 raft_consensus.cc:504] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.309618 16514 leader_election.cc:288] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f43f06c83e4479f930c371892a1c50a (127.4.177.65:37649), 4a7417011d154f81a783509b665e5dd6 (127.4.177.67:33535)
I20260808 21:06:51.312749 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.309882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:54839 (local address 127.4.177.65:37649)
0808 21:06:51.309956 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.309958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.309970 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:06:51.310161 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.310162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.310171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.310242 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:51.310436 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.310930 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.311976 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.312374 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.312465 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.312474 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.312628 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.312667 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:06:51.312776 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.309841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37649 (local address 127.4.177.66:54839)
0808 21:06:51.309887 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.309897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.309977 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.310264 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.310264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.310266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:51.310350 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.310352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.311013 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.311014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.311876 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.311879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.312461 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.312462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.312547 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.312551 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.312620 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.312668 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":29}
I20260808 21:06:51.312772 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.310091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:60095 (local address 127.4.177.67:33535)
0808 21:06:51.310164 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.310166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.310290 (+ 124us) server_negotiation.cc:407] Connection header received
0808 21:06:51.310361 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.310362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.310372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.310451 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:06:51.310610 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.311071 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.312045 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.312412 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.312496 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.312504 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.312654 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.312690 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 21:06:51.312788 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.309990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:33535 (local address 127.4.177.66:60095)
0808 21:06:51.310204 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.310213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:06:51.310307 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:06:51.310455 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:06:51.310455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:06:51.310457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:06:51.310531 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.310533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.311192 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.311193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.311946 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:06:51.311947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.312497 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.312497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:06:51.312568 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.312572 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:06:51.312650 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:06:51.312690 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":142,"threads_started":1}
I20260808 21:06:51.313203 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca336a1fa11a47609e1d05babd5f0b56" candidate_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f43f06c83e4479f930c371892a1c50a" is_pre_election: true
I20260808 21:06:51.313398 16147 raft_consensus.cc:2454] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4260a5d1454c75ba2ef2d999b55acc in term 0.
I20260808 21:06:51.313361 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca336a1fa11a47609e1d05babd5f0b56" candidate_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7417011d154f81a783509b665e5dd6" is_pre_election: true
I20260808 21:06:51.313499 16437 raft_consensus.cc:2454] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b4260a5d1454c75ba2ef2d999b55acc in term 0.
I20260808 21:06:51.313664 16223 leader_election.cc:302] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4260a5d1454c75ba2ef2d999b55acc, 4f43f06c83e4479f930c371892a1c50a; no voters:
I20260808 21:06:51.313825 16514 raft_consensus.cc:2781] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:06:51.313899 16514 raft_consensus.cc:482] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:06:51.313925 16514 raft_consensus.cc:3037] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:51.314687 16514 raft_consensus.cc:504] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.314829 16514 leader_election.cc:288] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [CANDIDATE]: Term 1 election: Requested vote from peers 4f43f06c83e4479f930c371892a1c50a (127.4.177.65:37649), 4a7417011d154f81a783509b665e5dd6 (127.4.177.67:33535)
I20260808 21:06:51.315011 16147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca336a1fa11a47609e1d05babd5f0b56" candidate_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f43f06c83e4479f930c371892a1c50a"
I20260808 21:06:51.315049 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca336a1fa11a47609e1d05babd5f0b56" candidate_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7417011d154f81a783509b665e5dd6"
I20260808 21:06:51.315096 16147 raft_consensus.cc:3037] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:51.315181 16437 raft_consensus.cc:3037] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:06:51.315914 16147 raft_consensus.cc:2454] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4260a5d1454c75ba2ef2d999b55acc in term 1.
I20260808 21:06:51.315923 16437 raft_consensus.cc:2454] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b4260a5d1454c75ba2ef2d999b55acc in term 1.
I20260808 21:06:51.316083 16223 leader_election.cc:302] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a7417011d154f81a783509b665e5dd6, 4b4260a5d1454c75ba2ef2d999b55acc; no voters:
I20260808 21:06:51.316172 16514 raft_consensus.cc:2781] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:06:51.316349 16514 raft_consensus.cc:686] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 1 LEADER]: Becoming Leader. State: Replica: 4b4260a5d1454c75ba2ef2d999b55acc, State: Running, Role: LEADER
I20260808 21:06:51.316448 16514 consensus_queue.cc:237] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } }
I20260808 21:06:51.317353 15989 catalog_manager.cc:5495] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b4260a5d1454c75ba2ef2d999b55acc (127.4.177.66). New cstate: current_term: 1 leader_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b4260a5d1454c75ba2ef2d999b55acc" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 36393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 } health_report { overall_health: UNKNOWN } } }
W20260808 21:06:51.318152 16345 tablet.cc:2369] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:06:51.328132 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.325805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.4.177.66:36393)
0808 21:06:51.325875 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:06:51.325877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:06:51.325885 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:06:51.325918 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:06:51.325919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:06:51.325936 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:06:51.325998 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:06:51.326109 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.326572 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.327379 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:06:51.327574 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:06:51.327643 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:06:51.327759 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:06:51.327857 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:06:51.327914 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:06:51.327948 (+ 34us) server_negotiation.cc:299] Negotiation successful
0808 21:06:51.327966 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:06:51.330072 16147 raft_consensus.cc:1261] T ca336a1fa11a47609e1d05babd5f0b56 P 4f43f06c83e4479f930c371892a1c50a [term 1 FOLLOWER]: Refusing update from remote peer 4b4260a5d1454c75ba2ef2d999b55acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:51.330065 16437 raft_consensus.cc:1261] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 1 FOLLOWER]: Refusing update from remote peer 4b4260a5d1454c75ba2ef2d999b55acc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:06:51.330449 16517 consensus_queue.cc:1035] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f43f06c83e4479f930c371892a1c50a" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:51.330602 16514 consensus_queue.cc:1035] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7417011d154f81a783509b665e5dd6" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:06:51.332975 16521 mvcc.cc:205] Tried to move back new op lower bound from 7316370273604673536 to 7316370273552809984. Current Snapshot: MvccSnapshot[applied={T|T < 7316370273604673536}]
I20260808 21:06:51.333542 16522 mvcc.cc:205] Tried to move back new op lower bound from 7316370273604673536 to 7316370273552809984. Current Snapshot: MvccSnapshot[applied={T|T < 7316370273604673536}]
I20260808 21:06:51.333621 16523 mvcc.cc:205] Tried to move back new op lower bound from 7316370273604673536 to 7316370273552809984. Current Snapshot: MvccSnapshot[applied={T|T < 7316370273604673536}]
I20260808 21:06:51.335597 15989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:51.335700 15989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:06:51.336447 15989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36351c4145e8448f93a585a35af25cb1: Sending DeleteTablet for 3 replicas of tablet ca336a1fa11a47609e1d05babd5f0b56
I20260808 21:06:51.336835 16271 tablet_service.cc:1514] Processing DeleteTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:51 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:40120
I20260808 21:06:51.336850 16416 tablet_service.cc:1514] Processing DeleteTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:51 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:41638
I20260808 21:06:51.337077 16125 tablet_service.cc:1514] Processing DeleteTablet for tablet ca336a1fa11a47609e1d05babd5f0b56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:06:51 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:49904
I20260808 21:06:51.337114 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 16057
I20260808 21:06:51.340335 16531 tablet_replica.cc:330] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: stopping tablet replica
I20260808 21:06:51.340416 16531 raft_consensus.cc:2229] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:06:51.340492 16531 raft_consensus.cc:2258] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:51.340682 16532 tablet_replica.cc:330] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: stopping tablet replica
I20260808 21:06:51.340787 16532 raft_consensus.cc:2229] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:06:51.340896 16532 raft_consensus.cc:2258] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:06:51.340996 16531 ts_tablet_manager.cc:1905] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:51.342713 16531 ts_tablet_manager.cc:1918] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:51.342780 16531 log.cc:1196] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/ca336a1fa11a47609e1d05babd5f0b56
I20260808 21:06:51.342986 16531 ts_tablet_manager.cc:1939] T ca336a1fa11a47609e1d05babd5f0b56 P 4b4260a5d1454c75ba2ef2d999b55acc: Deleting consensus metadata
I20260808 21:06:51.343477 15975 catalog_manager.cc:4841] TS 4b4260a5d1454c75ba2ef2d999b55acc (127.4.177.66:36393): tablet ca336a1fa11a47609e1d05babd5f0b56 (table test-table [id=64a0d4454a0a4c62a13dc4f0f1f2af09]) successfully deleted
I20260808 21:06:51.343815 16532 ts_tablet_manager.cc:1905] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:06:51.344936 16532 ts_tablet_manager.cc:1918] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:06:51.344986 16532 log.cc:1196] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/ca336a1fa11a47609e1d05babd5f0b56
I20260808 21:06:51.345181 16532 ts_tablet_manager.cc:1939] T ca336a1fa11a47609e1d05babd5f0b56 P 4a7417011d154f81a783509b665e5dd6: Deleting consensus metadata
I20260808 21:06:51.345598 15974 catalog_manager.cc:4841] TS 4a7417011d154f81a783509b665e5dd6 (127.4.177.67:33535): tablet ca336a1fa11a47609e1d05babd5f0b56 (table test-table [id=64a0d4454a0a4c62a13dc4f0f1f2af09]) successfully deleted
W20260808 21:06:51.356889 15974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:37649 (error 108)
I20260808 21:06:51.357549 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:06:51.357406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37649 (local address 127.0.0.1:49916)
0808 21:06:51.357465 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0808 21:06:51.357514 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 21:06:51.357615 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 16204
W20260808 21:06:51.357720 15974 catalog_manager.cc:4584] TS 4f43f06c83e4479f930c371892a1c50a (127.4.177.65:37649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca336a1fa11a47609e1d05babd5f0b56: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37649: connect: Connection refused (error 111)
I20260808 21:06:51.368551 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 16349
I20260808 21:06:51.374719 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 15953
2026-08-08T21:06:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2026 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[16537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[16537](info): set up 2 sockets
Aug 08 21:06:51 dist-test-slave-chsk krb5kdc[16537](info): commencing operation
krb5kdc: starting...
W20260808 21:06:52.472833 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:06:52 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:06:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangeradmin_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangeradmin_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangerlookup_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangerlookup_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
I20260808 21:06:52.650454 4805 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task496WfL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/postgres -l logfile start
2026-08-08 21:06:55.115 UTC [16569] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 21:06:55.115 UTC [16569] LOG: listening on IPv4 address "127.4.177.84", port 53337
2026-08-08 21:06:55.119 UTC [16569] LOG: listening on Unix socket "/tmp/.s.PGSQL.53337"
2026-08-08 21:06:55.141 UTC [16572] LOG: database system was shut down at 2026-08-08 21:06:53 UTC
2026-08-08 21:06:55.145 UTC [16569] LOG: database system is ready to accept connections
I20260808 21:06:56.133003 4805 mini_postgres.cc:96] Postgres bound to 53337
2026-08-08 21:06:56.141 UTC [16581] FATAL: database "slave" does not exist
127.4.177.84:53337 - accepting connections
I20260808 21:06:56.141877 4805 mini_ranger.cc:161] Starting Ranger...
I20260808 21:06:56.159716 4805 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 21:06:56.586825 4805 mini_ranger.cc:87] Created ranger Postgres database
I20260808 21:06:56.586913 4805 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin
2026-08-08 21:06:56,967 [I] DB FLAVOR :POSTGRES
2026-08-08 21:06:56,968 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 21:06:56,968 [I] Checking connection..
2026-08-08 21:06:56,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 21:06:57,221 [I] Checking connection passed.
2026-08-08 21:06:57,222 [I] --------- Verifying version history table ---------
2026-08-08 21:06:57,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:06:57,478 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 21:06:57,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:06:57,711 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 21:06:57,711 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 21:06:57,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 21:06:57.969 UTC [16651] WARNING: there is no transaction in progress
2026-08-08 21:06:57,976 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 21:06:57,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:06:58,242 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 21:06:58,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 21:06:58,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 21:06:58,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 21:06:58,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-chsk.c.gcp-upstream.internal','N') ;"
2026-08-08 21:06:59,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 21:06:59,267 [I] Table x_portal_user does not exist in database ranger
2026-08-08 21:06:59,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 21:06:59,531 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 21:06:59,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 21:06:59,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 21:07:00,027 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 21:07:00,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 21:07:00.971 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:00.986 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.001 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.016 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.031 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.048 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.112 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.119 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.132 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.142 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.151 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.164 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.174 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.180 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.186 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01.192 UTC [16771] WARNING: there is no transaction in progress
2026-08-08 21:07:01,800 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 21:07:01,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 21:07:02,051 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 21:07:02,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 21:07:02,310 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 21:07:02,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 21:07:02,571 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 21:07:02,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 21:07:02,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 21:07:03,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-chsk.c.gcp-upstream.internal';"
2026-08-08 21:07:03,377 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 21:07:03,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 21:07:03,639 [I] DB_PATCHES have already been applied
I20260808 21:07:03.645104 4805 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task496WfL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 21:07:03.645174 4805 mini_ranger.cc:193] Using host: 127.4.177.84
I20260808 21:07:03.649045 4805 mini_ranger.cc:227] Ranger admin URL: http://127.4.177.84:52587
Aug 08, 2026 9:07:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 9:07:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 9:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 9:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 9:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 9:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 9:07:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 21:07:04.819541 4805 mini_ranger.cc:160] Time spent starting Ranger: real 8.678s user 0.000s sys 0.012s
Aug 08, 2026 9:07:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 9:07:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 9:07:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 9:07:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 9:07:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 21:07:27.175645 4805 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangerkms_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/rangerkms_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.177.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.177.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 21:07:27 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223247, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 21:07:27.255342 4805 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 21:07:27.265139 4805 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 21:07:28.055429 4805 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 21:07:28.055501 4805 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms
I20260808 21:07:28.055514 4805 mini_ranger_kms.cc:227] Using postgres at 127.4.177.84:53337
2026-08-08 21:07:28,281 [I] DB FLAVOR :POSTGRES
2026-08-08 21:07:28,282 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 21:07:28,282 [I] Checking connection
2026-08-08 21:07:28,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 21:07:28,523 [I] connection success
2026-08-08 21:07:28,523 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 21:07:28,523 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 21:07:28,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 21:07:28,795 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 21:07:28,795 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 21:07:28,795 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 21:07:28,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:53337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 21:07:29,058 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 21:07:29.530010 4805 mini_ranger_kms.cc:310] Created kms service
I20260808 21:07:29.690459 4805 mini_ranger_kms.cc:326] Created kudu user
I20260808 21:07:29.737243 4805 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 21:07:29 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223247, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:30.135833 4805 mini_ranger_kms.cc:384] Added ranger policy
I20260808 21:07:30.135962 4805 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms:/tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 21:07:30.135986 4805 mini_ranger_kms.cc:240] Using host: 127.4.177.84
I20260808 21:07:30.140592 4805 mini_ranger_kms.cc:276] Ranger KMS PID: 16966
I20260808 21:07:30.140722 4805 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.177.84:37073
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 9:07:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 21:07:32.544713 4805 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.289s user 0.015s sys 0.195s
I20260808 21:07:32.544831 4805 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 21:07:32.544924 4805 mini_ranger_kms.cc:401] 127.4.177.84:37073/kms/v1/keys
Aug 08 21:07:32 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786223252, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 21:07:33 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223252, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
Aug 08 21:07:33 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223252, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
Aug 08 21:07:35 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:07:35.642884 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37419
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:37419
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:37073/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:07:35.725885 17022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:07:35.726094 17022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:07:35.726114 17022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:07:35.727233 17022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:07:35.727272 17022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:07:35.727286 17022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:07:35.727326 17022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:07:35.727355 17022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:07:35.729828 17022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.177.84:37073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:37419
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:07:35.730095 17022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:07:35.730343 17022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:07:35.736299 17029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:35.736373 17031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:35.736323 17028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:07:35.736629 17022 server_base.cc:1034] running on GCE node
I20260808 21:07:35.736891 17022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:07:35.737229 17022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:07:35.738456 17022 hybrid_clock.cc:648] HybridClock initialized: now 1786223255738415 us; error 53 us; skew 500 ppm
Aug 08 21:07:35 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:07:35.741545 17022 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:07:35.742034 17022 webserver.cc:460] Webserver started at http://127.4.177.126:38691/ using document root <none> and password file <none>
I20260808 21:07:35.742235 17022 fs_manager.cc:359] Metadata directory not provided
I20260808 21:07:35.742281 17022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:07:35.742372 17022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:07:35 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223255, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:35.791034 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "1d07a83848f1411eb516947245cf47f3"
format_stamp: "Formatted at 2026-08-08 21:07:35 on dist-test-slave-chsk"
server_key: "3c6914584a37951965e63a2e7ca570dc"
server_key_iv: "e6b478014bb8e170c6b239735c0d4457"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:35.791607 17022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "1d07a83848f1411eb516947245cf47f3"
format_stamp: "Formatted at 2026-08-08 21:07:35 on dist-test-slave-chsk"
server_key: "3c6914584a37951965e63a2e7ca570dc"
server_key_iv: "e6b478014bb8e170c6b239735c0d4457"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:35.793207 17022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:07:35.805855 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:35.806118 17022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:07:35.806221 17022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "1d07a83848f1411eb516947245cf47f3"
format_stamp: "Formatted at 2026-08-08 21:07:35 on dist-test-slave-chsk"
server_key: "3c6914584a37951965e63a2e7ca570dc"
server_key_iv: "e6b478014bb8e170c6b239735c0d4457"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:35.806322 17022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:07:35.819635 17022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:07:35.821771 17022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:07:35.821947 17022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:07:35.826776 17022 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:37419
I20260808 21:07:35.826830 17093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:37419 every 8 connection(s)
I20260808 21:07:35.827270 17022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:07:35.827955 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:07:35.829701 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 17022
I20260808 21:07:35.829813 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:07:35.830003 4805 external_mini_cluster.cc:1428] Setting key 16433e72601dbf334fcc1004568f5af6
I20260808 21:07:35.830749 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: Bootstrap starting.
I20260808 21:07:35.831490 17094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: Neither blocks nor log segments found. Creating new log.
I20260808 21:07:35.831868 17094 log.cc:824] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:07:35.832732 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: No bootstrap required, opened a new log
I20260808 21:07:35.835373 17094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } }
I20260808 21:07:35.835544 17094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:07:35.835577 17094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d07a83848f1411eb516947245cf47f3, State: Initialized, Role: FOLLOWER
I20260808 21:07:35.835732 17094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } }
I20260808 21:07:35.835809 17094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:07:35.835863 17094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:07:35.835913 17094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:07:35.836602 17094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } }
I20260808 21:07:35.836714 17094 leader_election.cc:302] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d07a83848f1411eb516947245cf47f3; no voters:
Aug 08 21:07:35 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:07:35.836915 17094 leader_election.cc:288] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:07:35.836979 17099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:07:35.837211 17094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:07:35.837195 17099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1d07a83848f1411eb516947245cf47f3, State: Running, Role: LEADER
I20260808 21:07:35.837399 17099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } }
I20260808 21:07:35.837769 17101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d07a83848f1411eb516947245cf47f3. Latest consensus state: current_term: 1 leader_uuid: "1d07a83848f1411eb516947245cf47f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } } }
I20260808 21:07:35.837958 17101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [sys.catalog]: This master's current role is: LEADER
I20260808 21:07:35.839305 17102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d07a83848f1411eb516947245cf47f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d07a83848f1411eb516947245cf47f3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37419 } } }
I20260808 21:07:35.839452 17102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3 [sys.catalog]: This master's current role is: LEADER
I20260808 21:07:35.839459 17094 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 21:07:35.838927 17104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:07:35.843861 17104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:07:35.844614 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:35.830551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.4.177.126:37419)
0808 21:07:35.830704 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:35.830714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:35.830748 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:07:35.831339 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:35.831348 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:35.831408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:35.831563 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:07:35.832340 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:35.833320 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:35.834101 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:35.834542 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:35.834680 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:35.837367 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:07:35.837372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:07:35.837372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:07:35.837386 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:07:35.839631 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:35.839994 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:35.839997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:35.839997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:35.840100 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:35.840313 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:35.840314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:35.840314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:35.842560 (+ 2246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:07:35.843360 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:35.843500 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 21:07:35.844154 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260808 21:07:35.848474 17104 catalog_manager.cc:1347] Generated new cluster ID: acbdac0be4304214b3c3342fc0f329fc
I20260808 21:07:35.848531 17104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:07:35.861121 17104 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:07:35.862996 17104 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:07:35.884804 17104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: Generated new TSK 0
I20260808 21:07:35.885136 17104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:07:36.611349 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:37073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:07:36.702880 17145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:07:36.703127 17145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:07:36.703164 17145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:07:36.704557 17145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:07:36.704630 17145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:07:36.704679 17145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:07:36.706205 17145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.177.84:37073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:07:36.706476 17145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:07:36.706725 17145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:07:36.712488 17150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:36.712488 17153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:36.712528 17151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:07:36.713057 17145 server_base.cc:1034] running on GCE node
I20260808 21:07:36.713258 17145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:07:36.713528 17145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:07:36.714682 17145 hybrid_clock.cc:648] HybridClock initialized: now 1786223256714673 us; error 28 us; skew 500 ppm
Aug 08 21:07:36 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:07:36.717365 17145 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:07:36.717882 17145 webserver.cc:460] Webserver started at http://127.4.177.65:40401/ using document root <none> and password file <none>
I20260808 21:07:36.718144 17145 fs_manager.cc:359] Metadata directory not provided
I20260808 21:07:36.718189 17145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:07:36.718321 17145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:07:36 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:36.732276 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "154d3439acc04bfbb7e884dc670f9a7f"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "e98b7dbcf6fb51e847355a8c224fd6a2"
server_key_iv: "1bbde4d26bc771384b5df83be91ac738"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:36.732772 17145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "154d3439acc04bfbb7e884dc670f9a7f"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "e98b7dbcf6fb51e847355a8c224fd6a2"
server_key_iv: "1bbde4d26bc771384b5df83be91ac738"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:36.734769 17145 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260808 21:07:36.747923 17160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:36.748209 17145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:07:36.748322 17145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "154d3439acc04bfbb7e884dc670f9a7f"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "e98b7dbcf6fb51e847355a8c224fd6a2"
server_key_iv: "1bbde4d26bc771384b5df83be91ac738"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:36.748441 17145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:07:36.760870 17145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:07:36.763060 17145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:07:36.763271 17145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:07:36.763567 17145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:07:36.763993 17145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:07:36.764056 17145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:36.764125 17145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:07:36.764171 17145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:36.771951 17145 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:33221
I20260808 21:07:36.772387 17145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:07:36.773051 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:33221 every 8 connection(s)
I20260808 21:07:36.777498 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 17145
I20260808 21:07:36.777604 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:07:36.777801 4805 external_mini_cluster.cc:1428] Setting key c3a15796dcd17bc26d1f70a60865fc88
Aug 08 21:07:36 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:07:36.788745 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:36.773223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:37031 (local address 127.4.177.126:37419)
0808 21:07:36.775311 (+ 2088us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:36.775313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:36.775322 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:07:36.775345 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:36.775346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:36.775356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:36.775445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:07:36.777026 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:36.777781 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:36.779277 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:36.779606 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:36.779649 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:36.782335 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:07:36.782336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:07:36.782336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:07:36.782344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:07:36.784499 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:36.785020 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:36.785021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:36.785022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:36.785103 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:36.785376 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:36.785377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:36.785377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:36.785516 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:07:36.785578 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:36.788582 (+ 3004us) server_negotiation.cc:299] Negotiation successful
0808 21:07:36.788671 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2058,"thread_start_us":97,"threads_started":1}
I20260808 21:07:36.788964 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:36.773207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37419 (local address 127.4.177.65:37031)
0808 21:07:36.773402 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:36.773446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:36.774095 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:36.775545 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:36.775550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:36.775756 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:07:36.775949 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:36.775953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:36.777902 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:36.777903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:36.779173 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:36.779176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:36.779743 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:36.779744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:36.779897 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:36.780528 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:07:36.780541 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:07:36.782211 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:07:36.784608 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:36.784616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:36.784618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:36.784933 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:36.785174 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:36.785176 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:36.785176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:36.785308 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:36.787298 (+ 1990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:07:36.787305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:07:36.787481 (+ 176us) client_negotiation.cc:769] Sending connection context
0808 21:07:36.788531 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0808 21:07:36.788719 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":120,"threads_started":1}
I20260808 21:07:36.789529 17283 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37419
I20260808 21:07:36.789654 17283 heartbeater.cc:464] Registering TS with master...
I20260808 21:07:36.789853 17283 heartbeater.cc:511] Master 127.4.177.126:37419 requested a full tablet report, sending...
I20260808 21:07:36.790438 17057 ts_manager.cc:194] Registered new tserver with Master: 154d3439acc04bfbb7e884dc670f9a7f (127.4.177.65:33221)
I20260808 21:07:36.791096 17057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:37031
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:07:36.821956 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:37073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:07:36.951653 17291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:07:36.952033 17291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:07:36.952126 17291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:07:36.954025 17291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:07:36.954279 17291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:07:36.954409 17291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:07:36.956863 17291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.177.84:37073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:07:36.957391 17291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:07:36.957744 17291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:07:36.964916 17298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:36.964951 17299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:36.965435 17301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:07:36.966601 17291 server_base.cc:1034] running on GCE node
I20260808 21:07:36.966956 17291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:07:36.967506 17291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:07:36.968746 17291 hybrid_clock.cc:648] HybridClock initialized: now 1786223256968722 us; error 37 us; skew 500 ppm
Aug 08 21:07:36 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:07:36.972671 17291 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:07:36.973245 17291 webserver.cc:460] Webserver started at http://127.4.177.66:33953/ using document root <none> and password file <none>
I20260808 21:07:36.973505 17291 fs_manager.cc:359] Metadata directory not provided
I20260808 21:07:36.973555 17291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:07:36.973686 17291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:07:36 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:36.995395 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "517dbe086ea44ae4beed5a82e586f770"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "fc426ae52a26537e1b8c013774a865c8"
server_key_iv: "daa05a8895b996d83eb6fb20ed5732e6"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:36.996084 17291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "517dbe086ea44ae4beed5a82e586f770"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "fc426ae52a26537e1b8c013774a865c8"
server_key_iv: "daa05a8895b996d83eb6fb20ed5732e6"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:37.001001 17291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260808 21:07:37.025646 17308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.025928 17291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260808 21:07:37.026078 17291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "517dbe086ea44ae4beed5a82e586f770"
format_stamp: "Formatted at 2026-08-08 21:07:36 on dist-test-slave-chsk"
server_key: "fc426ae52a26537e1b8c013774a865c8"
server_key_iv: "daa05a8895b996d83eb6fb20ed5732e6"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:37.026293 17291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786223257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:07:37.046972 17291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:07:37.049768 17291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:07:37.050026 17291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:07:37.050369 17291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:07:37.050820 17291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:07:37.050921 17291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.050995 17291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:07:37.051056 17291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.060814 17291 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:43887
I20260808 21:07:37.063511 17291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:07:37.064904 17430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:43887 every 8 connection(s)
I20260808 21:07:37.067970 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 17291
I20260808 21:07:37.068086 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:07:37.068295 4805 external_mini_cluster.cc:1428] Setting key d66840cf000c795431a62b1d5e824fe2
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223256, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:07:37.080749 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.067324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:49995 (local address 127.4.177.126:37419)
0808 21:07:37.067375 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.067377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.067386 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:07:37.067417 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.067418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.067429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.067503 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:07:37.071740 (+ 4237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.072264 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.073627 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.073912 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.074008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.076782 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:07:37.076784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:07:37.076784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:07:37.076793 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:07:37.078733 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.079379 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.079381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.079382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.079454 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.079774 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.079775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.079775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.079943 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:07:37.080004 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.080460 (+ 456us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.080578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 21:07:37.081694 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.065636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37419 (local address 127.4.177.66:49995)
0808 21:07:37.066078 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.066125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.066752 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.067610 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.067614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.068846 (+ 1232us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:07:37.071623 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.071632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.072391 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.072392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.073507 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.073510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.074014 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.074015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.074158 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.074801 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:07:37.074811 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:07:37.076676 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:07:37.078850 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:37.078858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:37.078861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:37.079272 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:37.079532 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:37.079533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:37.079533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:37.079692 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:37.080043 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:07:37.080050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:07:37.080266 (+ 216us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.081139 (+ 873us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.081375 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":143,"threads_started":1}
I20260808 21:07:37.082612 17431 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37419
I20260808 21:07:37.082726 17431 heartbeater.cc:464] Registering TS with master...
I20260808 21:07:37.082947 17431 heartbeater.cc:511] Master 127.4.177.126:37419 requested a full tablet report, sending...
I20260808 21:07:37.083501 17057 ts_manager.cc:194] Registered new tserver with Master: 517dbe086ea44ae4beed5a82e586f770 (127.4.177.66:43887)
I20260808 21:07:37.084038 17057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:49995
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:07:37.115739 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:37073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:07:37.235013 17436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:07:37.235337 17436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:07:37.235383 17436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:07:37.237018 17436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:07:37.237157 17436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:07:37.237247 17436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:07:37.239596 17436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.177.84:37073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:07:37.240093 17436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:07:37.240463 17436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:07:37.247481 17444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:37.247884 17446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:07:37.248809 17443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:07:37.248950 17436 server_base.cc:1034] running on GCE node
I20260808 21:07:37.249226 17436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:07:37.249529 17436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:07:37.250725 17436 hybrid_clock.cc:648] HybridClock initialized: now 1786223257250701 us; error 37 us; skew 500 ppm
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:07:37.254117 17436 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:07:37.254671 17436 webserver.cc:460] Webserver started at http://127.4.177.67:35775/ using document root <none> and password file <none>
I20260808 21:07:37.254954 17436 fs_manager.cc:359] Metadata directory not provided
I20260808 21:07:37.255019 17436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:07:37.255180 17436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:37.277367 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "63a239b5be594295b6a539ef8b213de7"
format_stamp: "Formatted at 2026-08-08 21:07:37 on dist-test-slave-chsk"
server_key: "05568dacbb0c7150695c305814975bee"
server_key_iv: "c55a38667524da80f5fdfa4c3ddb1ca9"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:37.278024 17436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "63a239b5be594295b6a539ef8b213de7"
format_stamp: "Formatted at 2026-08-08 21:07:37 on dist-test-slave-chsk"
server_key: "05568dacbb0c7150695c305814975bee"
server_key_iv: "c55a38667524da80f5fdfa4c3ddb1ca9"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:37.283643 17436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260808 21:07:37.303296 17453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.303898 17436 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20260808 21:07:37.304026 17436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "63a239b5be594295b6a539ef8b213de7"
format_stamp: "Formatted at 2026-08-08 21:07:37 on dist-test-slave-chsk"
server_key: "05568dacbb0c7150695c305814975bee"
server_key_iv: "c55a38667524da80f5fdfa4c3ddb1ca9"
server_key_version: "kuduclusterkey@0"
I20260808 21:07:37.304136 17436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:07:37.333078 17436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:07:37.335851 17436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:07:37.336087 17436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:07:37.336386 17436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:07:37.336872 17436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:07:37.336975 17436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.337059 17436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:07:37.337126 17436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:07:37.350179 17436 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:35179
I20260808 21:07:37.350787 17436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:07:37.351761 17578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:35179 every 8 connection(s)
I20260808 21:07:37.354619 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 17436
I20260808 21:07:37.354733 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:07:37.354931 4805 external_mini_cluster.cc:1428] Setting key 2f7ca78691265b7a43761a723ebd71c4
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:07:37.366667 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.352639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:51949 (local address 127.4.177.126:37419)
0808 21:07:37.352689 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.352692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.352976 (+ 284us) server_negotiation.cc:407] Connection header received
0808 21:07:37.353797 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.353798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.353812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.353847 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:07:37.356177 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.356972 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.358493 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.358786 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.358840 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.361887 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:07:37.361889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:07:37.361889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:07:37.361898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:07:37.363732 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.365362 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.365364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.365364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.365436 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.365819 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.365820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.365821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.365988 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:07:37.366066 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.366469 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.366565 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:07:37.367643 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.352598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37419 (local address 127.4.177.67:51949)
0808 21:07:37.352818 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.352874 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.353655 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.354053 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.354058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.354324 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:07:37.356057 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.356066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.357101 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.357102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.358380 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.358383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.358945 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.358946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.359108 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.359841 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:07:37.359852 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:07:37.361755 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:07:37.364869 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:37.364880 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:37.364884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:37.365255 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:37.365521 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:07:37.365522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:07:37.365522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:07:37.365704 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:07:37.366114 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:07:37.366122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:07:37.366308 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.367095 (+ 787us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.367318 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260808 21:07:37.368383 17579 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37419
I20260808 21:07:37.368491 17579 heartbeater.cc:464] Registering TS with master...
I20260808 21:07:37.368670 17579 heartbeater.cc:511] Master 127.4.177.126:37419 requested a full tablet report, sending...
I20260808 21:07:37.369191 17057 ts_manager.cc:194] Registered new tserver with Master: 63a239b5be594295b6a539ef8b213de7 (127.4.177.67:35179)
I20260808 21:07:37.369688 17057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:51949
I20260808 21:07:37.378975 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:07:37.389793 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.381217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.4.177.126:37419)
0808 21:07:37.381269 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.381271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.381554 (+ 283us) server_negotiation.cc:407] Connection header received
0808 21:07:37.381655 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.381656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.381667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.381711 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:07:37.382558 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.383377 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.384472 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.384790 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.384836 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.386641 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:07:37.386643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:07:37.386643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:07:37.386651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:07:37.388420 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.388871 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.388873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.388873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.388941 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:07:37.389179 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:07:37.389181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:07:37.389181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:07:37.389320 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:07:37.389420 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.389613 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.389699 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:07:37.391737 17057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:07:37.394057 17057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:07:37.403555 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.397408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:35179 (local address 127.0.0.1:37748)
0808 21:07:37.397671 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.397837 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.398034 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.398973 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.398977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.398980 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:07:37.399065 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.399067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.400703 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.400704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.402039 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.402043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.403201 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.403202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.403306 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.403322 (+ 16us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.403407 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.403464 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":112,"threads_started":1}
I20260808 21:07:37.403595 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.398292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.4.177.67:35179)
0808 21:07:37.398678 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.398684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.398696 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:07:37.398737 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.398739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.398756 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.398933 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:07:37.399222 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.400600 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.402180 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.403081 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.403193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.403297 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.403446 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.403493 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":133,"threads_started":1}
I20260808 21:07:37.404774 17508 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 (DEFAULT_TABLE table=test-table [id=2241cd3223b5482db04ef0ae8f317e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:07:37.405200 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed5f05d054a43cb8fd44a4aa15a0336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:07:37.408262 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.397861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33221 (local address 127.0.0.1:35080)
0808 21:07:37.403414 (+ 5553us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.403424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.403474 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.403950 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.403951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.403953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:07:37.404144 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.404146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.405491 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.405493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.406664 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.406666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.407956 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.407957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.408031 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.408036 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.408086 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.408174 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5511,"thread_start_us":177,"threads_started":1}
I20260808 21:07:37.408836 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.398487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.4.177.65:33221)
0808 21:07:37.399061 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.399066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.403720 (+ 4654us) server_negotiation.cc:407] Connection header received
0808 21:07:37.403766 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.403769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.403788 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.403871 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:07:37.404253 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.405384 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.406768 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.407799 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.407862 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.407921 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.408702 (+ 781us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.408749 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":76,"threads_started":1}
I20260808 21:07:37.409948 17206 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 (DEFAULT_TABLE table=test-table [id=2241cd3223b5482db04ef0ae8f317e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:07:37.410355 17206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed5f05d054a43cb8fd44a4aa15a0336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:07:37.414521 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.397417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:43887 (local address 127.0.0.1:54214)
0808 21:07:37.403227 (+ 5810us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.403240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.403297 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.407539 (+ 4242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.407541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.407543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:07:37.407585 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.407587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.408930 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.408931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.410127 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.410130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.414240 (+ 4110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.414242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.414317 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.414322 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.414389 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.414439 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5787,"thread_start_us":290,"threads_started":1}
I20260808 21:07:37.414932 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.398487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.4.177.66:43887)
0808 21:07:37.407279 (+ 8792us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.407284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.407296 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:07:37.407338 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.407340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.407357 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.407447 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:07:37.407670 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.408787 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.410270 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.410957 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.414238 (+ 3281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.414339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.414778 (+ 439us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.414829 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8743,"thread_start_us":170,"threads_started":1}
I20260808 21:07:37.414510 17599 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Bootstrap starting.
I20260808 21:07:37.415653 17358 tablet_service.cc:1467] Processing CreateTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 (DEFAULT_TABLE table=test-table [id=2241cd3223b5482db04ef0ae8f317e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:07:37.416087 17358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ed5f05d054a43cb8fd44a4aa15a0336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:07:37.416497 17599 tablet_bootstrap.cc:654] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Neither blocks nor log segments found. Creating new log.
I20260808 21:07:37.416851 17599 log.cc:824] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:07:37.417829 17599 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: No bootstrap required, opened a new log
I20260808 21:07:37.417992 17599 ts_tablet_manager.cc:1397] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 21:07:37.425158 17599 raft_consensus.cc:348] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.425383 17599 raft_consensus.cc:374] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:07:37.425414 17599 raft_consensus.cc:729] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a239b5be594295b6a539ef8b213de7, State: Initialized, Role: FOLLOWER
I20260808 21:07:37.425565 17599 consensus_queue.cc:260] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.425876 17599 ts_tablet_manager.cc:1428] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260808 21:07:37.425999 17579 heartbeater.cc:503] Master 127.4.177.126:37419 was elected leader, sending a full tablet report...
I20260808 21:07:37.428359 17600 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Bootstrap starting.
I20260808 21:07:37.429297 17600 tablet_bootstrap.cc:654] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Neither blocks nor log segments found. Creating new log.
I20260808 21:07:37.429649 17600 log.cc:824] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:07:37.430130 17602 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Bootstrap starting.
I20260808 21:07:37.431270 17602 tablet_bootstrap.cc:654] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Neither blocks nor log segments found. Creating new log.
I20260808 21:07:37.431615 17602 log.cc:824] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Log is configured to *not* fsync() on all Append() calls
I20260808 21:07:37.433260 17600 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: No bootstrap required, opened a new log
I20260808 21:07:37.433349 17600 ts_tablet_manager.cc:1397] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 21:07:37.436115 17600 raft_consensus.cc:348] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.436312 17600 raft_consensus.cc:374] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:07:37.436357 17600 raft_consensus.cc:729] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154d3439acc04bfbb7e884dc670f9a7f, State: Initialized, Role: FOLLOWER
I20260808 21:07:37.436518 17600 consensus_queue.cc:260] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.436864 17600 ts_tablet_manager.cc:1428] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 21:07:37.436998 17283 heartbeater.cc:503] Master 127.4.177.126:37419 was elected leader, sending a full tablet report...
I20260808 21:07:37.438328 17602 tablet_bootstrap.cc:492] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: No bootstrap required, opened a new log
I20260808 21:07:37.438438 17602 ts_tablet_manager.cc:1397] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260808 21:07:37.440069 17602 raft_consensus.cc:348] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.440255 17602 raft_consensus.cc:374] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:07:37.440306 17602 raft_consensus.cc:729] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517dbe086ea44ae4beed5a82e586f770, State: Initialized, Role: FOLLOWER
I20260808 21:07:37.440442 17602 consensus_queue.cc:260] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.440868 17602 ts_tablet_manager.cc:1428] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:07:37.440881 17431 heartbeater.cc:503] Master 127.4.177.126:37419 was elected leader, sending a full tablet report...
I20260808 21:07:37.516016 17604 raft_consensus.cc:482] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:07:37.516170 17604 raft_consensus.cc:504] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.516609 17604 leader_election.cc:288] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 517dbe086ea44ae4beed5a82e586f770 (127.4.177.66:43887), 154d3439acc04bfbb7e884dc670f9a7f (127.4.177.65:33221)
I20260808 21:07:37.520592 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.516793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:43887 (local address 127.4.177.67:35153)
0808 21:07:37.517065 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.517076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.517117 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.517490 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.517491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.517493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:07:37.517562 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.517565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.518517 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.518518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.519758 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.519761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.520346 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.520347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.520421 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.520424 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.520474 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.520516 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260808 21:07:37.521001 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.517333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:44367 (local address 127.4.177.65:33221)
0808 21:07:37.517377 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.517379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.517553 (+ 174us) server_negotiation.cc:407] Connection header received
0808 21:07:37.517656 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.517658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.517667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.517698 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:07:37.517910 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.518580 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.519948 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.520567 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.520663 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.520671 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.520887 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.520932 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:07:37.521061 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.517269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:35153 (local address 127.4.177.66:43887)
0808 21:07:37.517321 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.517323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.517332 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:07:37.517362 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.517363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.517372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.517431 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:07:37.517643 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.518414 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.519878 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.520255 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.520350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.520359 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.520951 (+ 592us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.520994 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:07:37.521351 17377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed5f05d054a43cb8fd44a4aa15a0336" candidate_uuid: "63a239b5be594295b6a539ef8b213de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "517dbe086ea44ae4beed5a82e586f770" is_pre_election: true
I20260808 21:07:37.521456 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.516884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33221 (local address 127.4.177.67:44367)
0808 21:07:37.517493 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.517503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:07:37.517575 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:07:37.517770 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:07:37.517771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:07:37.517772 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:07:37.517834 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.517835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.518664 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.518664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.519789 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:07:37.519791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.520713 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.520714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:07:37.520791 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.520796 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:07:37.521331 (+ 535us) client_negotiation.cc:240] Negotiation successful
0808 21:07:37.521377 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260808 21:07:37.521489 17377 raft_consensus.cc:2454] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a239b5be594295b6a539ef8b213de7 in term 0.
I20260808 21:07:37.521720 17454 leader_election.cc:302] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 517dbe086ea44ae4beed5a82e586f770, 63a239b5be594295b6a539ef8b213de7; no voters:
I20260808 21:07:37.521868 17604 raft_consensus.cc:2781] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:07:37.521938 17604 raft_consensus.cc:482] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:07:37.521963 17604 raft_consensus.cc:3037] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:07:37.521996 17232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed5f05d054a43cb8fd44a4aa15a0336" candidate_uuid: "63a239b5be594295b6a539ef8b213de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154d3439acc04bfbb7e884dc670f9a7f" is_pre_election: true
I20260808 21:07:37.522153 17232 raft_consensus.cc:2454] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63a239b5be594295b6a539ef8b213de7 in term 0.
I20260808 21:07:37.522921 17604 raft_consensus.cc:504] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.523065 17604 leader_election.cc:288] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [CANDIDATE]: Term 1 election: Requested vote from peers 517dbe086ea44ae4beed5a82e586f770 (127.4.177.66:43887), 154d3439acc04bfbb7e884dc670f9a7f (127.4.177.65:33221)
I20260808 21:07:37.523291 17232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed5f05d054a43cb8fd44a4aa15a0336" candidate_uuid: "63a239b5be594295b6a539ef8b213de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "154d3439acc04bfbb7e884dc670f9a7f"
I20260808 21:07:37.523311 17377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ed5f05d054a43cb8fd44a4aa15a0336" candidate_uuid: "63a239b5be594295b6a539ef8b213de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "517dbe086ea44ae4beed5a82e586f770"
I20260808 21:07:37.523367 17232 raft_consensus.cc:3037] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:07:37.523388 17377 raft_consensus.cc:3037] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 0 FOLLOWER]: Advancing to term 1
W20260808 21:07:37.523448 17284 tablet.cc:2369] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:07:37.524309 17377 raft_consensus.cc:2454] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a239b5be594295b6a539ef8b213de7 in term 1.
I20260808 21:07:37.524348 17232 raft_consensus.cc:2454] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63a239b5be594295b6a539ef8b213de7 in term 1.
I20260808 21:07:37.524588 17454 leader_election.cc:302] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 154d3439acc04bfbb7e884dc670f9a7f, 63a239b5be594295b6a539ef8b213de7; no voters:
I20260808 21:07:37.524675 17604 raft_consensus.cc:2781] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:07:37.524852 17604 raft_consensus.cc:686] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 1 LEADER]: Becoming Leader. State: Replica: 63a239b5be594295b6a539ef8b213de7, State: Running, Role: LEADER
I20260808 21:07:37.524959 17604 consensus_queue.cc:237] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } }
I20260808 21:07:37.525825 17056 catalog_manager.cc:5495] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63a239b5be594295b6a539ef8b213de7 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "63a239b5be594295b6a539ef8b213de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a239b5be594295b6a539ef8b213de7" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20260808 21:07:37.549875 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.546446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.4.177.67:35179)
0808 21:07:37.546512 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:07:37.546515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:07:37.546524 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:07:37.546565 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:07:37.546566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:07:37.546597 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:07:37.546679 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:07:37.546851 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.547637 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.548736 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:07:37.549098 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:07:37.549174 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:07:37.549401 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:07:37.549531 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:07:37.549616 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:07:37.549773 (+ 157us) server_negotiation.cc:299] Negotiation successful
0808 21:07:37.549792 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:07:37.553382 17377 raft_consensus.cc:1261] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 1 FOLLOWER]: Refusing update from remote peer 63a239b5be594295b6a539ef8b213de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:07:37.553411 17232 raft_consensus.cc:1261] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 154d3439acc04bfbb7e884dc670f9a7f [term 1 FOLLOWER]: Refusing update from remote peer 63a239b5be594295b6a539ef8b213de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:07:37.553748 17604 consensus_queue.cc:1035] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "154d3439acc04bfbb7e884dc670f9a7f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:07:37.554068 17604 consensus_queue.cc:1035] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "517dbe086ea44ae4beed5a82e586f770" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:07:37.557539 17613 mvcc.cc:205] Tried to move back new op lower bound from 7316370462935175168 to 7316370462822879232. Current Snapshot: MvccSnapshot[applied={T|T < 7316370462935175168}]
I20260808 21:07:37.560456 17056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:07:37.560557 17056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:07:37.561349 17056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d07a83848f1411eb516947245cf47f3: Sending DeleteTablet for 3 replicas of tablet 9ed5f05d054a43cb8fd44a4aa15a0336
I20260808 21:07:37.561699 17508 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:07:37 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:37748
I20260808 21:07:37.561872 17206 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:07:37 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:35080
I20260808 21:07:37.561913 17358 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ed5f05d054a43cb8fd44a4aa15a0336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:07:37 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54214
I20260808 21:07:37.562146 17623 tablet_replica.cc:330] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: stopping tablet replica
I20260808 21:07:37.562242 17623 raft_consensus.cc:2229] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:07:37.562333 17623 raft_consensus.cc:2258] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:07:37.562696 17614 mvcc.cc:205] Tried to move back new op lower bound from 7316370462935175168 to 7316370462822879232. Current Snapshot: MvccSnapshot[applied={T|T < 7316370462935175168}]
I20260808 21:07:37.567076 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 17145
I20260808 21:07:37.567214 17623 ts_tablet_manager.cc:1905] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:07:37.573268 17623 ts_tablet_manager.cc:1918] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:07:37.573383 17623 log.cc:1196] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/9ed5f05d054a43cb8fd44a4aa15a0336
I20260808 21:07:37.573652 17623 ts_tablet_manager.cc:1939] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 517dbe086ea44ae4beed5a82e586f770: Deleting consensus metadata
I20260808 21:07:37.574148 17040 catalog_manager.cc:4841] TS 517dbe086ea44ae4beed5a82e586f770 (127.4.177.66:43887): tablet 9ed5f05d054a43cb8fd44a4aa15a0336 (table test-table [id=2241cd3223b5482db04ef0ae8f317e8b]) successfully deleted
I20260808 21:07:37.577517 17621 tablet_replica.cc:330] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: stopping tablet replica
I20260808 21:07:37.577724 17621 raft_consensus.cc:2229] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:07:37.577906 17621 raft_consensus.cc:2258] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:07:37.578472 17621 ts_tablet_manager.cc:1905] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:07:37.580358 17621 ts_tablet_manager.cc:1918] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:07:37.580495 17621 log.cc:1196] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/9ed5f05d054a43cb8fd44a4aa15a0336
I20260808 21:07:37.580856 17621 ts_tablet_manager.cc:1939] T 9ed5f05d054a43cb8fd44a4aa15a0336 P 63a239b5be594295b6a539ef8b213de7: Deleting consensus metadata
I20260808 21:07:37.581393 17043 catalog_manager.cc:4841] TS 63a239b5be594295b6a539ef8b213de7 (127.4.177.67:35179): tablet 9ed5f05d054a43cb8fd44a4aa15a0336 (table test-table [id=2241cd3223b5482db04ef0ae8f317e8b]) successfully deleted
W20260808 21:07:37.586063 17040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:33221 (error 108)
I20260808 21:07:37.587023 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 17291
I20260808 21:07:37.587035 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:07:37.586884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:33221 (local address 127.0.0.1:35082)
0808 21:07:37.586942 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:07:37.586997 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260808 21:07:37.588163 17040 catalog_manager.cc:4584] TS 154d3439acc04bfbb7e884dc670f9a7f (127.4.177.65:33221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ed5f05d054a43cb8fd44a4aa15a0336: Network error: Client connection negotiation failed: client connection to 127.4.177.65:33221: connect: Connection refused (error 111)
Aug 08 21:07:37 dist-test-slave-chsk krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223257, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:07:37.594653 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 17436
I20260808 21:07:37.601778 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 17022
I20260808 21:07:37.610153 4805 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 21:07:37.955571 4805 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 21:07:37.955677 4805 mini_ranger.cc:66] Stopping Ranger...
I20260808 21:07:38.300499 4805 mini_ranger.cc:68] Stopped Ranger
2026-08-08 21:07:38.300 UTC [16569] LOG: received smart shutdown request
2026-08-08 21:07:38.304 UTC [16569] LOG: background worker "logical replication launcher" (PID 16579) exited with exit code 1
2026-08-08 21:07:38.305 UTC [16573] LOG: shutting down
2026-08-08 21:07:38.483 UTC [16569] LOG: database system is shut down
2026-08-08T21:07:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47223 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:07:38 dist-test-slave-chsk krb5kdc[17636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:07:38 dist-test-slave-chsk krb5kdc[17636](info): set up 2 sockets
Aug 08 21:07:38 dist-test-slave-chsk krb5kdc[17636](info): commencing operation
krb5kdc: starting...
W20260808 21:07:39.700134 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:07:39 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:07:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangeradmin_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangeradmin_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangerlookup_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangerlookup_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
I20260808 21:07:39.878031 4805 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task496WfL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/postgres -l logfile start
2026-08-08 21:07:42.435 UTC [17670] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 21:07:42.435 UTC [17670] LOG: listening on IPv4 address "127.4.177.84", port 60799
2026-08-08 21:07:42.439 UTC [17670] LOG: listening on Unix socket "/tmp/.s.PGSQL.60799"
2026-08-08 21:07:42.474 UTC [17673] LOG: database system was shut down at 2026-08-08 21:07:40 UTC
2026-08-08 21:07:42.480 UTC [17670] LOG: database system is ready to accept connections
I20260808 21:07:43.472000 4805 mini_postgres.cc:96] Postgres bound to 60799
2026-08-08 21:07:43.479 UTC [17681] FATAL: database "slave" does not exist
127.4.177.84:60799 - accepting connections
I20260808 21:07:43.479962 4805 mini_ranger.cc:161] Starting Ranger...
I20260808 21:07:43.495769 4805 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 21:07:43.999452 4805 mini_ranger.cc:87] Created ranger Postgres database
I20260808 21:07:43.999553 4805 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin
2026-08-08 21:07:44,279 [I] DB FLAVOR :POSTGRES
2026-08-08 21:07:44,279 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 21:07:44,279 [I] Checking connection..
2026-08-08 21:07:44,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 21:07:44,534 [I] Checking connection passed.
2026-08-08 21:07:44,535 [I] --------- Verifying version history table ---------
2026-08-08 21:07:44,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:07:44,810 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 21:07:44,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:07:45,071 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 21:07:45,071 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 21:07:45,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 21:07:45.322 UTC [17751] WARNING: there is no transaction in progress
2026-08-08 21:07:45,330 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 21:07:45,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 21:07:45,562 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 21:07:45,563 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 21:07:45,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 21:07:45,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 21:07:46,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-chsk.c.gcp-upstream.internal','N') ;"
2026-08-08 21:07:46,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 21:07:46,569 [I] Table x_portal_user does not exist in database ranger
2026-08-08 21:07:46,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 21:07:46,836 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 21:07:46,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 21:07:47,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 21:07:47,340 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 21:07:47,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 21:07:48.305 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.322 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.337 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.352 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.367 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.384 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.440 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.446 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.461 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.471 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.481 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.495 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.505 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.511 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.517 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:48.524 UTC [17868] WARNING: there is no transaction in progress
2026-08-08 21:07:49,129 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 21:07:49,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 21:07:49,363 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 21:07:49,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 21:07:49,626 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 21:07:49,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 21:07:49,885 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 21:07:49,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 21:07:50,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 21:07:50,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-chsk.c.gcp-upstream.internal';"
2026-08-08 21:07:50,662 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 21:07:50,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 21:07:50,927 [I] DB_PATCHES have already been applied
I20260808 21:07:50.932948 4805 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task496WfL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 21:07:50.933048 4805 mini_ranger.cc:193] Using host: 127.4.177.84
I20260808 21:07:50.936892 4805 mini_ranger.cc:227] Ranger admin URL: http://127.4.177.84:54039
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 9:07:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 21:07:52.090058 4805 mini_ranger.cc:160] Time spent starting Ranger: real 8.610s user 0.003s sys 0.008s
Aug 08, 2026 9:08:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 9:08:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 9:08:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 9:08:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 9:08:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 21:08:13.935621 4805 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.177.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.177.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangerkms_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.177.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/rangerkms_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.177.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.177.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.177.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/HTTP_127.4.177.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 21:08:14 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 21:08:14.016687 4805 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 21:08:14.026189 4805 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 21:08:14.847100 4805 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 21:08:14.847240 4805 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms
I20260808 21:08:14.847261 4805 mini_ranger_kms.cc:227] Using postgres at 127.4.177.84:60799
2026-08-08 21:08:15,017 [I] DB FLAVOR :POSTGRES
2026-08-08 21:08:15,018 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 21:08:15,018 [I] Checking connection
2026-08-08 21:08:15,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 21:08:15,248 [I] connection success
2026-08-08 21:08:15,249 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 21:08:15,249 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 21:08:15,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 21:08:15,509 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 21:08:15,509 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 21:08:15,509 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 21:08:15,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.177.84:60799/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 21:08:15,781 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 21:08:16.258992 4805 mini_ranger_kms.cc:310] Created kms service
I20260808 21:08:16.487006 4805 mini_ranger_kms.cc:326] Created kudu user
I20260808 21:08:16.530354 4805 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 21:08:16 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:16.933604 4805 mini_ranger_kms.cc:384] Added ranger policy
I20260808 21:08:16.933954 4805 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms:/tmp/dist-test-task496WfL/build/release/bin/postgresql.jar:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task496WfL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 21:08:16.933988 4805 mini_ranger_kms.cc:240] Using host: 127.4.177.84
I20260808 21:08:16.938120 4805 mini_ranger_kms.cc:276] Ranger KMS PID: 18058
I20260808 21:08:16.938200 4805 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.177.84:47179
Aug 08, 2026 9:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 9:08:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 9:08:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260808 21:08:19.254284 4805 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.238s user 0.002s sys 0.151s
I20260808 21:08:19.254395 4805 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 21:08:19.254446 4805 mini_ranger_kms.cc:401] 127.4.177.84:47179/kms/v1/keys
Aug 08 21:08:19 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786223299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 21:08:19 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
Aug 08 21:08:20 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.177.84@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
Aug 08 21:08:22 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:08:22.309669 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42373
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42373
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:22.416914 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:22.417148 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:22.417187 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:22.418313 18112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:22.418376 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:22.418402 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:22.418434 18112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:22.418466 18112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:22.420123 18112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.177.84:47179/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42373
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:22.420404 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:22.420639 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:22.426357 18119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:22.426365 18118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:22.426399 18121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:22.426700 18112 server_base.cc:1034] running on GCE node
I20260808 21:08:22.426973 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:22.427304 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:22.428488 18112 hybrid_clock.cc:648] HybridClock initialized: now 1786223302428465 us; error 41 us; skew 500 ppm
Aug 08 21:08:22 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:22.434624 18112 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:22.435174 18112 webserver.cc:460] Webserver started at http://127.4.177.126:44593/ using document root <none> and password file <none>
I20260808 21:08:22.435438 18112 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:22.435500 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:22.435662 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:08:22 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223302, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:22.482997 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "ba552f2d3e4f4f1ab866ea3a51c43852"
format_stamp: "Formatted at 2026-08-08 21:08:22 on dist-test-slave-chsk"
server_key: "2805439cb8608cff2189e0350eeda899"
server_key_iv: "90520d7e99fd9f0ae52c8d1c6baf27ff"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:22.483526 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "ba552f2d3e4f4f1ab866ea3a51c43852"
format_stamp: "Formatted at 2026-08-08 21:08:22 on dist-test-slave-chsk"
server_key: "2805439cb8608cff2189e0350eeda899"
server_key_iv: "90520d7e99fd9f0ae52c8d1c6baf27ff"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:22.484997 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:22.496476 18128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:22.496749 18112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:22.496873 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "ba552f2d3e4f4f1ab866ea3a51c43852"
format_stamp: "Formatted at 2026-08-08 21:08:22 on dist-test-slave-chsk"
server_key: "2805439cb8608cff2189e0350eeda899"
server_key_iv: "90520d7e99fd9f0ae52c8d1c6baf27ff"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:22.496982 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:22.506680 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:22.508894 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:22.509048 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:22.514030 18112 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42373
I20260808 21:08:22.514109 18186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42373 every 8 connection(s)
I20260808 21:08:22.514472 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:22.515229 18187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:22.516503 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18112
I20260808 21:08:22.516629 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:08:22.516839 4805 external_mini_cluster.cc:1428] Setting key 022f69b6924aa6d50ba3ca1f24c782b3
I20260808 21:08:22.518005 18187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Bootstrap starting.
I20260808 21:08:22.518921 18187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:22.519358 18187 log.cc:824] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:22.520164 18187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: No bootstrap required, opened a new log
I20260808 21:08:22.522471 18187 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:22.522711 18187 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:22.522778 18187 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba552f2d3e4f4f1ab866ea3a51c43852, State: Initialized, Role: FOLLOWER
I20260808 21:08:22.522948 18187 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:22.523046 18187 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:22.523108 18187 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:22.523195 18187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:22.523945 18187 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:22.524082 18187 leader_election.cc:302] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba552f2d3e4f4f1ab866ea3a51c43852; no voters:
I20260808 21:08:22.524246 18187 leader_election.cc:288] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:08:22.524313 18193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:22.524528 18193 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 LEADER]: Becoming Leader. State: Replica: ba552f2d3e4f4f1ab866ea3a51c43852, State: Running, Role: LEADER
I20260808 21:08:22.524627 18187 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:22.524669 18193 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:22.525058 18193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba552f2d3e4f4f1ab866ea3a51c43852. Latest consensus state: current_term: 1 leader_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } } }
Aug 08 21:08:22 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:22.525071 18195 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } } }
I20260808 21:08:22.525131 18193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:22.525308 18195 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:22.525499 18197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:22.526357 18197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:22.526775 18187 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 21:08:22.528373 18211 mvcc.cc:205] Tried to move back new op lower bound from 7316370647153295360 to 7316370647141523456. Current Snapshot: MvccSnapshot[applied={T|T < 7316370647153295360}]
I20260808 21:08:22.528870 18197 catalog_manager.cc:1347] Generated new cluster ID: a6afdebc16324c34977e1f5f7d2e92a1
I20260808 21:08:22.528916 18197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:08:22.529840 18190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:22.517369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.4.177.126:42373)
0808 21:08:22.517650 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:22.517660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:22.517693 (+ 33us) server_negotiation.cc:407] Connection header received
0808 21:08:22.518210 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:22.518218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:22.518282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:22.518431 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:22.519389 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:22.520818 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:22.521980 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:22.522607 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:22.522731 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:22.525708 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:22.525713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:22.525713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:22.525726 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:22.527888 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:22.528287 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:22.528290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:22.528291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:22.528384 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:22.528598 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:22.528599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:22.528599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:22.528881 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:22.529016 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:22.529504 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 21:08:22.529649 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260808 21:08:22.544195 18197 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:08:22.545096 18197 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:22.553599 18197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Generated new TSK 0
I20260808 21:08:22.553864 18197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:08:23.298882 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:23.420269 18235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:23.420539 18235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:23.420572 18235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:23.422168 18235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:23.422231 18235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:23.422271 18235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:23.424391 18235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:23.424644 18235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:23.424918 18235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:23.430258 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:23.430270 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:23.430263 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:23.432355 18235 server_base.cc:1034] running on GCE node
I20260808 21:08:23.432583 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:23.432902 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:23.434095 18235 hybrid_clock.cc:648] HybridClock initialized: now 1786223303434063 us; error 45 us; skew 500 ppm
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:23.437201 18235 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:23.437681 18235 webserver.cc:460] Webserver started at http://127.4.177.65:46845/ using document root <none> and password file <none>
I20260808 21:08:23.437934 18235 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:23.438007 18235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:23.438158 18235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:23.457074 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "a169b552e3794a1da34fa7efb075a60f"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "4d5b2bf794986b492feb371a3fd1a027"
server_key_iv: "32fc504e668a347fbb31a10fffcc964c"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.458197 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "a169b552e3794a1da34fa7efb075a60f"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "4d5b2bf794986b492feb371a3fd1a027"
server_key_iv: "32fc504e668a347fbb31a10fffcc964c"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.460155 18235 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:23.478730 18250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.479002 18235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260808 21:08:23.479126 18235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "a169b552e3794a1da34fa7efb075a60f"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "4d5b2bf794986b492feb371a3fd1a027"
server_key_iv: "32fc504e668a347fbb31a10fffcc964c"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.479250 18235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:23.519315 18235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:23.521749 18235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:23.521898 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:23.522151 18235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:23.522505 18235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:23.522552 18235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.522583 18235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:23.522641 18235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.529980 18235 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:42661
I20260808 21:08:23.530117 18381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:42661 every 8 connection(s)
I20260808 21:08:23.530486 18235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:23.539301 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18235
I20260808 21:08:23.539412 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:08:23.539618 4805 external_mini_cluster.cc:1428] Setting key 677101ddbeb2416305c11d3015fb8a0d
I20260808 21:08:23.540740 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:23.531393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.65:45971)
0808 21:08:23.531570 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:23.531611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:23.532079 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:23.532342 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:23.532345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:23.532516 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:23.532698 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:23.532701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.533705 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.533706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:23.534468 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:23.534470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.534961 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.534961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:23.535052 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:23.535632 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:23.535640 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:23.537166 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:23.538790 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:23.538796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:23.538798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:23.539154 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:23.539437 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:23.539438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:23.539438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:23.539580 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:23.540085 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:23.540092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:23.540254 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 21:08:23.540362 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 21:08:23.540512 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260808 21:08:23.541384 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:23.531423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:45971 (local address 127.4.177.126:42373)
0808 21:08:23.531583 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:23.531585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:23.531704 (+ 119us) server_negotiation.cc:407] Connection header received
0808 21:08:23.532175 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:23.532177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:23.532192 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:23.532295 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:23.532788 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.533609 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.534558 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.534857 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.534941 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:23.537267 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:23.537268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:23.537268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:23.537275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:23.538706 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:23.539248 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:23.539250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:23.539250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:23.539323 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:23.539668 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:23.539669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:23.539670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:23.539986 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:23.540047 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:23.541191 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0808 21:08:23.541284 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260808 21:08:23.542281 18382 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:23.542374 18382 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:23.542563 18382 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:23.543210 18147 ts_manager.cc:194] Registered new tserver with Master: a169b552e3794a1da34fa7efb075a60f (127.4.177.65:42661)
I20260808 21:08:23.544109 18147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:45971
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:08:23.569782 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 21:08:23.690833 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:23.691190 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:23.691277 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:23.692986 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:23.693092 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:23.693185 18388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:23.695412 18388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:23.695665 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:23.695976 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:23.702585 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:23.702936 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:23.703194 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:23.704388 18388 server_base.cc:1034] running on GCE node
I20260808 21:08:23.704692 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:23.704947 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:23.706133 18388 hybrid_clock.cc:648] HybridClock initialized: now 1786223303706106 us; error 40 us; skew 500 ppm
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:23.709583 18388 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:23.710100 18388 webserver.cc:460] Webserver started at http://127.4.177.66:36211/ using document root <none> and password file <none>
I20260808 21:08:23.710428 18388 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:23.710543 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:23.710734 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:23.725118 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "f9ad600871a54b108e8cd86079301da3"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "86fb1d8a8b46290d6450f01b10e7bf3f"
server_key_iv: "6bf990102fba8fdc68cc8027b28ebfd5"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.725565 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "f9ad600871a54b108e8cd86079301da3"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "86fb1d8a8b46290d6450f01b10e7bf3f"
server_key_iv: "6bf990102fba8fdc68cc8027b28ebfd5"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.727917 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260808 21:08:23.743295 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.743561 18388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 21:08:23.743659 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "f9ad600871a54b108e8cd86079301da3"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "86fb1d8a8b46290d6450f01b10e7bf3f"
server_key_iv: "6bf990102fba8fdc68cc8027b28ebfd5"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:23.743749 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:23.770777 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:23.773489 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:23.773643 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:23.773883 18388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:23.774268 18388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:23.774318 18388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.774349 18388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:23.774366 18388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:23.781186 18388 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:35755
I20260808 21:08:23.781662 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:08:23.784286 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18388
I20260808 21:08:23.784397 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:08:23.784638 4805 external_mini_cluster.cc:1428] Setting key acd137a0a16c03274e7ada313acd9515
I20260808 21:08:23.792351 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:35755 every 8 connection(s)
Aug 08 21:08:23 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:23.804785 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:23.793433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:59447 (local address 127.4.177.126:42373)
0808 21:08:23.793479 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:23.793481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:23.794471 (+ 990us) server_negotiation.cc:407] Connection header received
0808 21:08:23.795232 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:23.795233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:23.795249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:23.795278 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:23.796143 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.796901 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.798351 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.798672 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.798719 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:23.801546 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:23.801548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:23.801548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:23.801568 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:23.803290 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:23.803749 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:23.803750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:23.803750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:23.803818 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:23.804076 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:23.804076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:23.804077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:23.804196 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:23.804233 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:23.804604 (+ 371us) server_negotiation.cc:299] Negotiation successful
0808 21:08:23.804691 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:08:23.805624 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:23.793127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.66:59447)
0808 21:08:23.794326 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:23.794378 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:23.795071 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:23.795495 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:23.795500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:23.795763 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:23.796030 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:23.796036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.797027 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.797027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:23.798245 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:23.798248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:23.798827 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:23.798828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:23.798969 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:23.799694 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:23.799704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:23.801424 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:23.803384 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:23.803390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:23.803393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:23.803662 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:23.803883 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:23.803884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:23.803884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:23.804007 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:23.804278 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:23.804284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:23.804450 (+ 166us) client_negotiation.cc:769] Sending connection context
0808 21:08:23.805133 (+ 683us) client_negotiation.cc:240] Negotiation successful
0808 21:08:23.805318 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":107,"threads_started":1}
I20260808 21:08:23.806324 18530 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:23.806423 18530 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:23.806641 18530 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:23.807127 18147 ts_manager.cc:194] Registered new tserver with Master: f9ad600871a54b108e8cd86079301da3 (127.4.177.66:35755)
I20260808 21:08:23.807641 18147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:59447
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:08:23.831276 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:24.001533 18535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:24.001888 18535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:24.001935 18535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:24.003741 18535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:24.003859 18535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:24.003944 18535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:24.006105 18535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:24.006544 18535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:24.006855 18535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:24.013324 18541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.013713 18540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.014326 18543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.014627 18535 server_base.cc:1034] running on GCE node
I20260808 21:08:24.014988 18535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:24.015281 18535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:24.016451 18535 hybrid_clock.cc:648] HybridClock initialized: now 1786223304016430 us; error 33 us; skew 500 ppm
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:24.019752 18535 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:24.020226 18535 webserver.cc:460] Webserver started at http://127.4.177.67:45105/ using document root <none> and password file <none>
I20260808 21:08:24.020467 18535 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:24.020514 18535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:24.020632 18535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:24.045658 18535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "0950b9a0564e4b50a426c7c413fd09dc"
format_stamp: "Formatted at 2026-08-08 21:08:24 on dist-test-slave-chsk"
server_key: "417415c4a66dc471306bf647fc6e2384"
server_key_iv: "35db26393c93e974036c4dfac4e48a50"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.046130 18535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "0950b9a0564e4b50a426c7c413fd09dc"
format_stamp: "Formatted at 2026-08-08 21:08:24 on dist-test-slave-chsk"
server_key: "417415c4a66dc471306bf647fc6e2384"
server_key_iv: "35db26393c93e974036c4dfac4e48a50"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.055402 18535 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260808 21:08:24.073922 18550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.074193 18535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260808 21:08:24.074283 18535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "0950b9a0564e4b50a426c7c413fd09dc"
format_stamp: "Formatted at 2026-08-08 21:08:24 on dist-test-slave-chsk"
server_key: "417415c4a66dc471306bf647fc6e2384"
server_key_iv: "35db26393c93e974036c4dfac4e48a50"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.074364 18535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:24.108006 18535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:24.111213 18535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:24.111445 18535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:24.111788 18535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:24.112286 18535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:24.112394 18535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.112478 18535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:24.112540 18535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.121872 18535 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46595
I20260808 21:08:24.122521 18535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:08:24.123054 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46595 every 8 connection(s)
I20260808 21:08:24.124426 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18535
I20260808 21:08:24.124544 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:08:24.124804 4805 external_mini_cluster.cc:1428] Setting key 6b5e3fee8c47ee5b1a41dc6dd64409ae
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:24.138121 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.125224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:40627 (local address 127.4.177.126:42373)
0808 21:08:24.125372 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:24.125375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:24.126441 (+ 1066us) server_negotiation.cc:407] Connection header received
0808 21:08:24.127368 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:24.127369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:24.127381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:24.127412 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:24.128379 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.129124 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.130599 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.130898 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.130945 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.133756 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:24.133757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:24.133758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:24.133765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:24.135749 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.136954 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.136955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.136956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.137023 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.137329 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.137330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.137331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.137474 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:24.137522 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:24.137930 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 21:08:24.138026 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260808 21:08:24.138934 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.123991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.67:40627)
0808 21:08:24.126316 (+ 2325us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:24.126360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:24.127239 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:24.127622 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:24.127627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:24.127927 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:24.128265 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.128272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.129248 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.129249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.130479 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.130482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.131069 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.131070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.131273 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.131878 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:24.131888 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:24.133620 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:24.135853 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.135860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.135862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.136861 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.137103 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.137104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.137104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.137238 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.137587 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:24.137593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:24.137761 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 21:08:24.138493 (+ 732us) client_negotiation.cc:240] Negotiation successful
0808 21:08:24.138693 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2281,"thread_start_us":96,"threads_started":1}
I20260808 21:08:24.139649 18673 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:24.139745 18673 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.139931 18673 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:24.140347 18147 ts_manager.cc:194] Registered new tserver with Master: 0950b9a0564e4b50a426c7c413fd09dc (127.4.177.67:46595)
I20260808 21:08:24.140844 18147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:40627
I20260808 21:08:24.147650 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:08:24.148399 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18235
I20260808 21:08:24.167742 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18388
I20260808 21:08:24.177435 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18535
I20260808 21:08:24.190644 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18112
I20260808 21:08:24.198930 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42373
--webserver_interface=127.4.177.126
--webserver_port=44593
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42373
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:24.303105 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:24.303390 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:24.303429 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:24.304544 18676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:24.304610 18676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:24.304636 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:24.304666 18676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:24.304713 18676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:24.306207 18676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42373
--ranger_config_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=44593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:24.306459 18676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:24.306694 18676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:24.312686 18684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.312738 18682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.312942 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.312963 18676 server_base.cc:1034] running on GCE node
I20260808 21:08:24.313254 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:24.313576 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:24.314754 18676 hybrid_clock.cc:648] HybridClock initialized: now 1786223304314730 us; error 38 us; skew 500 ppm
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:24.317904 18676 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:24.318471 18676 webserver.cc:460] Webserver started at http://127.4.177.126:44593/ using document root <none> and password file <none>
I20260808 21:08:24.318708 18676 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:24.318776 18676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:24.395608 18676 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.396449 18692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.396659 18676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.396775 18676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "ba552f2d3e4f4f1ab866ea3a51c43852"
format_stamp: "Formatted at 2026-08-08 21:08:22 on dist-test-slave-chsk"
server_key: "2805439cb8608cff2189e0350eeda899"
server_key_iv: "90520d7e99fd9f0ae52c8d1c6baf27ff"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.397140 18676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:24.419389 18676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:24.421298 18676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:24.421469 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:24.425597 18676 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42373
I20260808 21:08:24.425650 18750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42373 every 8 connection(s)
I20260808 21:08:24.425997 18676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:24.426574 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18676
I20260808 21:08:24.426892 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:42661
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:42373
--webserver_port=46845
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:24.427410 18751 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:08:24.428087 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Bootstrap starting.
I20260808 21:08:24.436273 18751 log.cc:824] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:24.437994 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:24.438293 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Bootstrap complete.
I20260808 21:08:24.440810 18751 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:24.441074 18751 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba552f2d3e4f4f1ab866ea3a51c43852, State: Initialized, Role: FOLLOWER
I20260808 21:08:24.441267 18751 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:24.441344 18751 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:24.441401 18751 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:24.441452 18751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:08:24.442454 18751 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:24.442574 18751 leader_election.cc:302] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba552f2d3e4f4f1ab866ea3a51c43852; no voters:
I20260808 21:08:24.442788 18751 leader_election.cc:288] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 21:08:24.442822 18755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:08:24.442936 18755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [term 2 LEADER]: Becoming Leader. State: Replica: ba552f2d3e4f4f1ab866ea3a51c43852, State: Running, Role: LEADER
I20260808 21:08:24.443074 18751 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:24.443261 18755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } }
I20260808 21:08:24.444674 18751 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 21:08:24.446862 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba552f2d3e4f4f1ab866ea3a51c43852. Latest consensus state: current_term: 2 leader_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } } }
I20260808 21:08:24.447201 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:24.446861 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba552f2d3e4f4f1ab866ea3a51c43852" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42373 } } }
I20260808 21:08:24.447302 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:24.447657 18775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:24.448450 18775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:24.448727 18775 catalog_manager.cc:1259] Loaded cluster ID: a6afdebc16324c34977e1f5f7d2e92a1
I20260808 21:08:24.448817 18775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:08:24.450428 18775 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:24.450691 18775 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Loaded TSK: 0
I20260808 21:08:24.451006 18775 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 21:08:24.538455 18753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:24.538727 18753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:24.538769 18753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:24.540357 18753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:24.540418 18753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:24.540457 18753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:24.542601 18753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:42661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=46845
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:24.542973 18753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:24.543282 18753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:24.549666 18791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.549932 18753 server_base.cc:1034] running on GCE node
W20260808 21:08:24.550405 18794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.549764 18792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.551282 18753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:24.551550 18753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:24.556092 18753 hybrid_clock.cc:648] HybridClock initialized: now 1786223304556059 us; error 43 us; skew 500 ppm
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:24.559376 18753 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:24.559933 18753 webserver.cc:460] Webserver started at http://127.4.177.65:46845/ using document root <none> and password file <none>
I20260808 21:08:24.560246 18753 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:24.560372 18753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:24.577808 18753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.579059 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.579270 18753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.579419 18753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "a169b552e3794a1da34fa7efb075a60f"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "4d5b2bf794986b492feb371a3fd1a027"
server_key_iv: "32fc504e668a347fbb31a10fffcc964c"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.579964 18753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:24.593847 18753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:24.596413 18753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:24.596632 18753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:24.596940 18753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:24.597359 18753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:24.597448 18753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.597551 18753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:24.597632 18753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.605307 18753 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:42661
I20260808 21:08:24.605398 18924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:42661 every 8 connection(s)
I20260808 21:08:24.605835 18753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:24.614908 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18753
I20260808 21:08:24.615257 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:35755
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:42373
--webserver_port=36211
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:24.622448 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.606573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.65:57637)
0808 21:08:24.606814 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:24.606854 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:24.607489 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:24.608037 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:24.608041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:24.608280 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:24.608544 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.608551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.610264 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.610265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.611551 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.611554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.612587 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.612588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.612720 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.613344 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:24.613354 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:24.617490 (+ 4136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:24.620090 (+ 2600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.620100 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.620102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.620618 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.621020 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.621021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.621021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.621178 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.621709 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:24.621716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:24.621874 (+ 158us) client_negotiation.cc:769] Sending connection context
0808 21:08:24.621976 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 21:08:24.622189 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260808 21:08:24.623090 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.606658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:57637 (local address 127.4.177.126:42373)
0808 21:08:24.607491 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:24.607499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:24.607584 (+ 85us) server_negotiation.cc:407] Connection header received
0808 21:08:24.607733 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:24.607740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:24.607793 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:24.607959 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:24.608695 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.610133 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.611649 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.612349 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.612473 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.617655 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:24.617660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:24.617660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:24.617673 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:24.619973 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.620809 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.620813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.620813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.620938 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.621259 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.621260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.621260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.621589 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:24.621676 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:24.622739 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0808 21:08:24.622872 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":93,"threads_started":1}
I20260808 21:08:24.624161 18926 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:24.624284 18926 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.624522 18926 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:24.625479 18926 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:42373 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 21:08:24.625566 18926 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.625630 18926 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:24.625989 18926 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.626077 18926 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:24.626395 18926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:08:24.748486 18930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:24.748812 18930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:24.748857 18930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:24.750669 18930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:24.750794 18930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:24.750886 18930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:24.753197 18930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:35755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=36211
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:24.753685 18930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:24.754045 18930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:24.760334 18935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.760329 18936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.760440 18938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.761130 18930 server_base.cc:1034] running on GCE node
I20260808 21:08:24.761361 18930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:24.761606 18930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:24.762773 18930 hybrid_clock.cc:648] HybridClock initialized: now 1786223304762760 us; error 31 us; skew 500 ppm
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:24.765612 18930 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:24.766103 18930 webserver.cc:460] Webserver started at http://127.4.177.66:36211/ using document root <none> and password file <none>
I20260808 21:08:24.766346 18930 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:24.766429 18930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:24.783530 18930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.791548 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.791802 18930 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260808 21:08:24.791914 18930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "f9ad600871a54b108e8cd86079301da3"
format_stamp: "Formatted at 2026-08-08 21:08:23 on dist-test-slave-chsk"
server_key: "86fb1d8a8b46290d6450f01b10e7bf3f"
server_key_iv: "6bf990102fba8fdc68cc8027b28ebfd5"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.792335 18930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:24.810791 18930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:24.813150 18930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:24.813323 18930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:24.813611 18930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:24.813992 18930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:24.814049 18930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.814119 18930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:24.814149 18930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.821933 18930 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:35755
I20260808 21:08:24.822063 19072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:35755 every 8 connection(s)
I20260808 21:08:24.822428 18930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:08:24.826088 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 18930
I20260808 21:08:24.826400 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.177.84:47179/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:46595
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:42373
--webserver_port=45105
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:24.839490 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.823373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:35659 (local address 127.4.177.126:42373)
0808 21:08:24.823452 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:24.823454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:24.823740 (+ 286us) server_negotiation.cc:407] Connection header received
0808 21:08:24.824453 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:24.824454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:24.824468 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:24.824526 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:24.825306 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.826009 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.827534 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.827860 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.827938 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.836204 (+ 8266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:24.836206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:24.836206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:24.836216 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:24.837733 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.838255 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.838257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.838257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.838321 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:24.838640 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:24.838641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:24.838641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:24.838790 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:24.838838 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:24.839290 (+ 452us) server_negotiation.cc:299] Negotiation successful
0808 21:08:24.839389 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:08:24.840281 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:24.823348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.66:35659)
0808 21:08:24.823570 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:24.823622 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:24.824325 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:24.824611 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:24.824616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:24.824883 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:24.825190 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.825196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.826123 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.826125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.827433 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:24.827436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:24.827986 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:24.827987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:24.828146 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:24.828699 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:24.828709 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:24.836070 (+ 7361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:24.837874 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.837883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.837885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.838151 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.838403 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:24.838404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:24.838404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:24.838550 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:24.838888 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:24.838895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:24.839083 (+ 188us) client_negotiation.cc:769] Sending connection context
0808 21:08:24.839828 (+ 745us) client_negotiation.cc:240] Negotiation successful
0808 21:08:24.840011 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260808 21:08:24.840994 19073 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:24.841122 19073 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.841320 19073 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:24.841985 19073 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:42373 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 21:08:24.842087 19073 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.842161 19073 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:24.842442 19073 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:24.842517 19073 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:24.842799 19073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:08:24.951778 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:24.952145 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:24.952235 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:24.954002 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:24.954079 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:24.954126 19077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:24.955821 19077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.177.84:47179/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:46595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=45105
--tserver_master_addrs=127.4.177.126:42373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:24.956111 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:24.956336 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:24.962423 19085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:24.962419 19083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.962646 19077 server_base.cc:1034] running on GCE node
W20260808 21:08:24.962738 19087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:24.962953 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:24.963238 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:24.964407 19077 hybrid_clock.cc:648] HybridClock initialized: now 1786223304964392 us; error 27 us; skew 500 ppm
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:24.967216 19077 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:24.967651 19077 webserver.cc:460] Webserver started at http://127.4.177.67:45105/ using document root <none> and password file <none>
I20260808 21:08:24.967913 19077 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:24.968019 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 21:08:24 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:24.985212 19077 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260808 21:08:24.986322 19097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:24.986526 19077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:24.986605 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "0950b9a0564e4b50a426c7c413fd09dc"
format_stamp: "Formatted at 2026-08-08 21:08:24 on dist-test-slave-chsk"
server_key: "417415c4a66dc471306bf647fc6e2384"
server_key_iv: "35db26393c93e974036c4dfac4e48a50"
server_key_version: "kuduclusterkey@0"
I20260808 21:08:24.987057 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:24.999941 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:25.001876 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:25.002049 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:25.002274 19077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:25.002590 19077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:25.002643 19077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:25.002686 19077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:25.002714 19077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:25.010291 19077 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46595
I20260808 21:08:25.015568 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46595 every 8 connection(s)
I20260808 21:08:25.016094 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:08:25 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:25.024422 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19077
I20260808 21:08:25.026117 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:25.016922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:46129 (local address 127.4.177.126:42373)
0808 21:08:25.016975 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:25.016978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:25.017248 (+ 270us) server_negotiation.cc:407] Connection header received
0808 21:08:25.017766 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:25.017767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:25.017781 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:25.017828 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:25.018506 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.018996 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.020025 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.020228 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.020304 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:25.022507 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:25.022509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:25.022509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:25.022517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:25.024434 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:25.024956 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:25.024957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:25.024957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:25.025020 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:25.025270 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:25.025271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:25.025271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:25.025439 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:25.025508 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:25.025905 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 21:08:25.025994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:08:25.026597 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:25.016936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42373 (local address 127.4.177.67:46129)
0808 21:08:25.017101 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:25.017143 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:25.017644 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:25.017895 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:25.017899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:25.018155 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:25.018409 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:25.018414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.019092 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.019093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:25.019931 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:25.019933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.020334 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.020336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:25.020474 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:25.021053 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:25.021073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:25.022393 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:25.024552 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:25.024559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:25.024561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:25.024853 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:25.025089 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:25.025090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:25.025090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:25.025191 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:25.025539 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:25.025547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:25.025728 (+ 181us) client_negotiation.cc:769] Sending connection context
0808 21:08:25.026256 (+ 528us) client_negotiation.cc:240] Negotiation successful
0808 21:08:25.026416 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260808 21:08:25.027042 19220 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42373
I20260808 21:08:25.027179 19220 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:25.027359 19220 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:25.028021 19220 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:42373 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 21:08:25.028121 19220 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:25.028210 19220 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:25.028528 19220 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:25.028611 19220 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
W20260808 21:08:25.028926 19220 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 21:08:25.036403 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:25.024889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.4.177.126:42373)
0808 21:08:25.025211 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:25.025213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:25.025222 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:25.025243 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:25.025244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:25.025257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:25.025326 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:25.026231 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.026962 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.028734 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:25.029113 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:25.029185 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:25.030452 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:25.030453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:25.030453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:25.030460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:25.035173 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:25.035573 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:25.035574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:25.035574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:25.035653 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:25.035867 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:25.035868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:25.035868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:25.035984 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:25.036162 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:25.036209 (+ 47us) server_negotiation.cc:299] Negotiation successful
0808 21:08:25.036289 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":61,"threads_started":1}
Aug 08 21:08:25 dist-test-slave-chsk krb5kdc[17636](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786223305, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:25.626595 18926 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:25.626755 18926 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:25.627254 18711 ts_manager.cc:194] Registered new tserver with Master: a169b552e3794a1da34fa7efb075a60f (127.4.177.65:42661)
I20260808 21:08:25.628250 18711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:57637
I20260808 21:08:25.842978 19073 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:25.843199 19073 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:25.843603 18711 ts_manager.cc:194] Registered new tserver with Master: f9ad600871a54b108e8cd86079301da3 (127.4.177.66:35755)
I20260808 21:08:25.844197 18711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:35659
Aug 08 21:08:25 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223305, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:26.029134 19220 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:26.029315 19220 heartbeater.cc:511] Master 127.4.177.126:42373 requested a full tablet report, sending...
I20260808 21:08:26.029592 18711 ts_manager.cc:194] Registered new tserver with Master: 0950b9a0564e4b50a426c7c413fd09dc (127.4.177.67:46595)
I20260808 21:08:26.030084 18711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:46129
I20260808 21:08:26.040678 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:08:26.053175 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.041688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.4.177.126:42373)
0808 21:08:26.041817 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.041819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.043864 (+ 2045us) server_negotiation.cc:407] Connection header received
0808 21:08:26.043913 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.043914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.043927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.044003 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:26.044678 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.045553 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.047283 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.047605 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.047672 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.049478 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:26.049479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:26.049479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:26.049487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:26.051482 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:26.051945 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:26.051946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:26.051947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:26.052018 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:26.052221 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:26.052222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:26.052222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:26.052364 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:26.052459 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.052942 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.053038 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260808 21:08:26.056041 18711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:08:26.058507 18711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:08:26.066672 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46595 (local address 127.0.0.1:37718)
0808 21:08:26.061586 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.061627 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:26.061700 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:26.062574 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:26.062577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:26.062579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:26.062650 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.062652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.063981 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.063982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.065363 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.065365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.066362 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.066363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.066437 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.066449 (+ 12us) client_negotiation.cc:769] Sending connection context
0808 21:08:26.066517 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:08:26.066573 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":27,"thread_start_us":92,"threads_started":1}
I20260808 21:08:26.067242 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.4.177.67:46595)
0808 21:08:26.062341 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.062346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.062416 (+ 70us) server_negotiation.cc:407] Connection header received
0808 21:08:26.062449 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.062451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.062468 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.062732 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:26.062815 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.063876 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.065457 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.066209 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.066278 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.066332 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.067089 (+ 757us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.067132 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":66,"threads_started":1}
I20260808 21:08:26.067885 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 (DEFAULT_TABLE table=test-table [id=440f13c4c82e49469a4d3bd1265e7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:26.068257 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aa5a19c4224ff6b62a3e58bebe3c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:26.068828 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.4.177.65:42661)
0808 21:08:26.064550 (+ 2839us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.064554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.064681 (+ 127us) server_negotiation.cc:407] Connection header received
0808 21:08:26.064781 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.064783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.064797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.064846 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:26.065045 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.066061 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.067666 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.068384 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.068433 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.068483 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.068712 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.068758 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2803,"thread_start_us":57,"threads_started":1}
I20260808 21:08:26.069036 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:42661 (local address 127.0.0.1:58532)
0808 21:08:26.064621 (+ 3023us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.064630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:26.064704 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:26.064937 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:26.064938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:26.064939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:26.064977 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.064978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.066163 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.066164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.067566 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.067568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.068546 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.068547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.068614 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.068618 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:08:26.068909 (+ 291us) client_negotiation.cc:240] Negotiation successful
0808 21:08:26.068955 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3004,"thread_start_us":71,"threads_started":1}
I20260808 21:08:26.069730 18852 tablet_service.cc:1467] Processing CreateTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 (DEFAULT_TABLE table=test-table [id=440f13c4c82e49469a4d3bd1265e7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:26.070077 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aa5a19c4224ff6b62a3e58bebe3c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:26.072064 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:35755 (local address 127.0.0.1:41458)
0808 21:08:26.062061 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.062072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:26.062150 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:26.062609 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:26.062610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:26.062611 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:26.062651 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.062652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.063973 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.063974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.070674 (+ 6700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.070677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.071829 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.071830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.071892 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.071896 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:08:26.071947 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 21:08:26.071992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":49,"thread_start_us":89,"threads_started":1}
I20260808 21:08:26.072360 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.061883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.4.177.66:35755)
0808 21:08:26.062064 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.062070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.062120 (+ 50us) server_negotiation.cc:407] Connection header received
0808 21:08:26.062409 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.062412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.062432 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.062510 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:26.062746 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.063886 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.070780 (+ 6894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.071655 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.071736 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.071789 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.072243 (+ 454us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.072288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260808 21:08:26.073087 18997 tablet_service.cc:1467] Processing CreateTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 (DEFAULT_TABLE table=test-table [id=440f13c4c82e49469a4d3bd1265e7bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:26.073438 18997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76aa5a19c4224ff6b62a3e58bebe3c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:26.073709 19256 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Bootstrap starting.
I20260808 21:08:26.074436 19256 tablet_bootstrap.cc:654] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:26.074816 19256 log.cc:824] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:26.075613 19256 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: No bootstrap required, opened a new log
I20260808 21:08:26.075673 19256 ts_tablet_manager.cc:1397] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:08:26.078228 19256 raft_consensus.cc:348] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.078413 19256 raft_consensus.cc:374] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:26.078457 19256 raft_consensus.cc:729] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a169b552e3794a1da34fa7efb075a60f, State: Initialized, Role: FOLLOWER
I20260808 21:08:26.078652 19256 consensus_queue.cc:260] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.078972 19256 ts_tablet_manager.cc:1428] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:08:26.079166 18926 heartbeater.cc:503] Master 127.4.177.126:42373 was elected leader, sending a full tablet report...
I20260808 21:08:26.079295 19255 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Bootstrap starting.
I20260808 21:08:26.079312 19258 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Bootstrap starting.
I20260808 21:08:26.079968 19258 tablet_bootstrap.cc:654] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:26.079999 19255 tablet_bootstrap.cc:654] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:26.080258 19258 log.cc:824] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:26.080299 19255 log.cc:824] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:26.081182 19258 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: No bootstrap required, opened a new log
I20260808 21:08:26.081257 19258 ts_tablet_manager.cc:1397] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:08:26.083812 19258 raft_consensus.cc:348] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.083976 19258 raft_consensus.cc:374] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:26.084007 19258 raft_consensus.cc:729] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ad600871a54b108e8cd86079301da3, State: Initialized, Role: FOLLOWER
I20260808 21:08:26.084174 19258 consensus_queue.cc:260] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.084453 19258 ts_tablet_manager.cc:1428] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:08:26.084487 19073 heartbeater.cc:503] Master 127.4.177.126:42373 was elected leader, sending a full tablet report...
I20260808 21:08:26.087944 19255 tablet_bootstrap.cc:492] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: No bootstrap required, opened a new log
I20260808 21:08:26.088024 19255 ts_tablet_manager.cc:1397] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260808 21:08:26.090581 19255 raft_consensus.cc:348] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.090775 19255 raft_consensus.cc:374] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:26.090809 19255 raft_consensus.cc:729] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0950b9a0564e4b50a426c7c413fd09dc, State: Initialized, Role: FOLLOWER
I20260808 21:08:26.090958 19255 consensus_queue.cc:260] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.091298 19255 ts_tablet_manager.cc:1428] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:08:26.091421 19220 heartbeater.cc:503] Master 127.4.177.126:42373 was elected leader, sending a full tablet report...
W20260808 21:08:26.108371 18927 tablet.cc:2369] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 21:08:26 dist-test-slave-chsk krb5kdc[17636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223305, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for HTTP/127.4.177.84@KRBTEST.COM
I20260808 21:08:26.251878 19262 raft_consensus.cc:482] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:26.252061 19262 raft_consensus.cc:504] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.252542 19262 leader_election.cc:288] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0950b9a0564e4b50a426c7c413fd09dc (127.4.177.67:46595), a169b552e3794a1da34fa7efb075a60f (127.4.177.65:42661)
I20260808 21:08:26.256626 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.253259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:42053 (local address 127.4.177.67:46595)
0808 21:08:26.253316 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.253318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.253326 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:08:26.253354 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.253355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.253364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.253409 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:26.253643 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.254335 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.255726 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.256240 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.256288 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.256296 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.256512 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.256548 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 21:08:26.256882 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.252781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46595 (local address 127.4.177.66:42053)
0808 21:08:26.253137 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.253151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:26.253203 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:26.253461 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:26.253462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:26.253464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:26.253550 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.253552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.254428 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.254429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.255625 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.255628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.256377 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.256378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.256439 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.256443 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:08:26.256747 (+ 304us) client_negotiation.cc:240] Negotiation successful
0808 21:08:26.256790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":112,"threads_started":1}
I20260808 21:08:26.257019 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.253417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:52531 (local address 127.4.177.65:42661)
0808 21:08:26.253473 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.253476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.253567 (+ 91us) server_negotiation.cc:407] Connection header received
0808 21:08:26.253675 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.253676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.253686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.253717 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:26.253940 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.254656 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.256031 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.256606 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.256659 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.256666 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.256897 (+ 231us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.256940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:08:26.257445 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.253001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:42661 (local address 127.4.177.66:52531)
0808 21:08:26.253496 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.253507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:26.253594 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:26.253785 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:26.253785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:26.253787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:26.253856 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.253858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.254765 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.254765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.255920 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:26.255924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.256733 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.256734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:26.256818 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.256821 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:26.257328 (+ 507us) client_negotiation.cc:240] Negotiation successful
0808 21:08:26.257370 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":67,"threads_started":1}
I20260808 21:08:26.257476 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aa5a19c4224ff6b62a3e58bebe3c92" candidate_uuid: "f9ad600871a54b108e8cd86079301da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0950b9a0564e4b50a426c7c413fd09dc" is_pre_election: true
I20260808 21:08:26.257628 19162 raft_consensus.cc:2454] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ad600871a54b108e8cd86079301da3 in term 0.
I20260808 21:08:26.257737 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aa5a19c4224ff6b62a3e58bebe3c92" candidate_uuid: "f9ad600871a54b108e8cd86079301da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a169b552e3794a1da34fa7efb075a60f" is_pre_election: true
I20260808 21:08:26.257885 18854 raft_consensus.cc:2454] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ad600871a54b108e8cd86079301da3 in term 0.
I20260808 21:08:26.258186 18949 leader_election.cc:302] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a169b552e3794a1da34fa7efb075a60f, f9ad600871a54b108e8cd86079301da3; no voters:
I20260808 21:08:26.258386 19262 raft_consensus.cc:2781] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:08:26.258440 19262 raft_consensus.cc:482] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:26.258466 19262 raft_consensus.cc:3037] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:26.259451 19262 raft_consensus.cc:504] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.259591 19262 leader_election.cc:288] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [CANDIDATE]: Term 1 election: Requested vote from peers 0950b9a0564e4b50a426c7c413fd09dc (127.4.177.67:46595), a169b552e3794a1da34fa7efb075a60f (127.4.177.65:42661)
I20260808 21:08:26.259842 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aa5a19c4224ff6b62a3e58bebe3c92" candidate_uuid: "f9ad600871a54b108e8cd86079301da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a169b552e3794a1da34fa7efb075a60f"
I20260808 21:08:26.259912 18854 raft_consensus.cc:3037] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:26.260871 18854 raft_consensus.cc:2454] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ad600871a54b108e8cd86079301da3 in term 1.
I20260808 21:08:26.261045 19162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76aa5a19c4224ff6b62a3e58bebe3c92" candidate_uuid: "f9ad600871a54b108e8cd86079301da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0950b9a0564e4b50a426c7c413fd09dc"
I20260808 21:08:26.261125 19162 raft_consensus.cc:3037] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:26.261946 19162 raft_consensus.cc:2454] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ad600871a54b108e8cd86079301da3 in term 1.
I20260808 21:08:26.262365 18949 leader_election.cc:302] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0950b9a0564e4b50a426c7c413fd09dc, f9ad600871a54b108e8cd86079301da3; no voters:
I20260808 21:08:26.262472 19262 raft_consensus.cc:2781] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:26.262609 19262 raft_consensus.cc:686] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 1 LEADER]: Becoming Leader. State: Replica: f9ad600871a54b108e8cd86079301da3, State: Running, Role: LEADER
I20260808 21:08:26.262701 19262 consensus_queue.cc:237] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } }
I20260808 21:08:26.263571 18711 catalog_manager.cc:5495] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9ad600871a54b108e8cd86079301da3 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "f9ad600871a54b108e8cd86079301da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9ad600871a54b108e8cd86079301da3" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 35755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 } health_report { overall_health: UNKNOWN } } }
W20260808 21:08:26.267561 19221 tablet.cc:2369] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:08:26.292459 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.288642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.4.177.66:35755)
0808 21:08:26.288691 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:26.288693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:26.288702 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:26.288748 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:26.288748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:26.288766 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:26.288856 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:08:26.289408 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.290124 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.291341 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:26.291655 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:26.291718 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:26.291905 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:08:26.292035 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:08:26.292106 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:26.292359 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 21:08:26.292380 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:08:26.297251 18854 raft_consensus.cc:1261] T 76aa5a19c4224ff6b62a3e58bebe3c92 P a169b552e3794a1da34fa7efb075a60f [term 1 FOLLOWER]: Refusing update from remote peer f9ad600871a54b108e8cd86079301da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:26.297711 19268 consensus_queue.cc:1035] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a169b552e3794a1da34fa7efb075a60f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:26.300192 19162 raft_consensus.cc:1261] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 1 FOLLOWER]: Refusing update from remote peer f9ad600871a54b108e8cd86079301da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:26.300585 19268 consensus_queue.cc:1035] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0950b9a0564e4b50a426c7c413fd09dc" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:26.304831 19271 mvcc.cc:205] Tried to move back new op lower bound from 7316370662589968384 to 7316370662452445184. Current Snapshot: MvccSnapshot[applied={T|T < 7316370662589968384}]
I20260808 21:08:26.305382 19274 mvcc.cc:205] Tried to move back new op lower bound from 7316370662589968384 to 7316370662452445184. Current Snapshot: MvccSnapshot[applied={T|T < 7316370662589968384}]
I20260808 21:08:26.307747 19272 mvcc.cc:205] Tried to move back new op lower bound from 7316370662589968384 to 7316370662452445184. Current Snapshot: MvccSnapshot[applied={T|T < 7316370662589968384}]
I20260808 21:08:26.310257 18711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:26.310348 18711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:26.311193 18711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba552f2d3e4f4f1ab866ea3a51c43852: Sending DeleteTablet for 3 replicas of tablet 76aa5a19c4224ff6b62a3e58bebe3c92
I20260808 21:08:26.311614 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:26 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:37718
I20260808 21:08:26.311733 18852 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:26 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:58532
I20260808 21:08:26.311932 18997 tablet_service.cc:1514] Processing DeleteTablet for tablet 76aa5a19c4224ff6b62a3e58bebe3c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:26 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:41458
I20260808 21:08:26.312611 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18753
I20260808 21:08:26.323240 19284 tablet_replica.cc:330] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: stopping tablet replica
I20260808 21:08:26.323371 19284 raft_consensus.cc:2229] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:08:26.323508 19284 raft_consensus.cc:2258] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:26.323683 19281 tablet_replica.cc:330] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: stopping tablet replica
I20260808 21:08:26.323799 19281 raft_consensus.cc:2229] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:26.323925 19281 raft_consensus.cc:2258] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:26.324193 19284 ts_tablet_manager.cc:1905] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:26.324340 19281 ts_tablet_manager.cc:1905] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:26.326232 19281 ts_tablet_manager.cc:1918] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:26.326316 19281 log.cc:1196] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/76aa5a19c4224ff6b62a3e58bebe3c92
I20260808 21:08:26.326612 19281 ts_tablet_manager.cc:1939] T 76aa5a19c4224ff6b62a3e58bebe3c92 P 0950b9a0564e4b50a426c7c413fd09dc: Deleting consensus metadata
W20260808 21:08:26.327037 18694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:42661 (error 108)
I20260808 21:08:26.327095 19284 ts_tablet_manager.cc:1918] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:26.327193 19284 log.cc:1196] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/76aa5a19c4224ff6b62a3e58bebe3c92
I20260808 21:08:26.327345 18694 catalog_manager.cc:4841] TS 0950b9a0564e4b50a426c7c413fd09dc (127.4.177.67:46595): tablet 76aa5a19c4224ff6b62a3e58bebe3c92 (table test-table [id=440f13c4c82e49469a4d3bd1265e7bf7]) successfully deleted
I20260808 21:08:26.327477 19284 ts_tablet_manager.cc:1939] T 76aa5a19c4224ff6b62a3e58bebe3c92 P f9ad600871a54b108e8cd86079301da3: Deleting consensus metadata
I20260808 21:08:26.327857 18695 catalog_manager.cc:4841] TS f9ad600871a54b108e8cd86079301da3 (127.4.177.66:35755): tablet 76aa5a19c4224ff6b62a3e58bebe3c92 (table test-table [id=440f13c4c82e49469a4d3bd1265e7bf7]) successfully deleted
I20260808 21:08:26.328076 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18930
I20260808 21:08:26.334246 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:26.334120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:42661 (local address 127.0.0.1:58540)
0808 21:08:26.334166 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:26.334203 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:42661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260808 21:08:26.334408 18694 catalog_manager.cc:4584] TS a169b552e3794a1da34fa7efb075a60f (127.4.177.65:42661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76aa5a19c4224ff6b62a3e58bebe3c92: Network error: Client connection negotiation failed: client connection to 127.4.177.65:42661: connect: Connection refused (error 111)
I20260808 21:08:26.340664 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19077
I20260808 21:08:26.350809 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 18676
I20260808 21:08:26.358871 4805 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 21:08:26.703724 4805 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 21:08:26.703843 4805 mini_ranger.cc:66] Stopping Ranger...
I20260808 21:08:27.048439 4805 mini_ranger.cc:68] Stopped Ranger
2026-08-08 21:08:27.048 UTC [17670] LOG: received smart shutdown request
2026-08-08 21:08:27.053 UTC [17670] LOG: background worker "logical replication launcher" (PID 17679) exited with exit code 1
2026-08-08 21:08:27.053 UTC [17674] LOG: shutting down
2026-08-08 21:08:27.211 UTC [17670] LOG: database system is shut down
2026-08-08T21:08:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48730 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:08:27 dist-test-slave-chsk krb5kdc[19296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:08:27 dist-test-slave-chsk krb5kdc[19296](info): set up 2 sockets
Aug 08 21:08:27 dist-test-slave-chsk krb5kdc[19296](info): commencing operation
krb5kdc: starting...
W20260808 21:08:28.443709 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:08:28.564011 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34667
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:28.645397 19311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:28.645586 19311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:28.645607 19311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:28.646725 19311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:28.646770 19311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:28.646785 19311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:28.646797 19311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:28.646809 19311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:28.648427 19311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:28.648669 19311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:28.648890 19311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:28.653607 19320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:28.653664 19317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:28.653831 19311 server_base.cc:1034] running on GCE node
W20260808 21:08:28.653913 19318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:28.654166 19311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:28.654467 19311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:28.655624 19311 hybrid_clock.cc:648] HybridClock initialized: now 1786223308655623 us; error 37 us; skew 500 ppm
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:28.661077 19311 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:28.661504 19311 webserver.cc:460] Webserver started at http://127.4.177.126:35925/ using document root <none> and password file <none>
I20260808 21:08:28.661715 19311 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:28.661756 19311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:28.661942 19311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:28.662946 19311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "1a64f3cf06ec4d92bf6afda23364f1c0"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "f8965cbe134db0851aa1476b5e42173f"
server_key_iv: "82840285f5c5ccc7660eaa2f06bffbcb"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.663352 19311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "1a64f3cf06ec4d92bf6afda23364f1c0"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "f8965cbe134db0851aa1476b5e42173f"
server_key_iv: "82840285f5c5ccc7660eaa2f06bffbcb"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.664791 19311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:08:28.665553 19327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:28.665724 19311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:08:28.665827 19311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "1a64f3cf06ec4d92bf6afda23364f1c0"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "f8965cbe134db0851aa1476b5e42173f"
server_key_iv: "82840285f5c5ccc7660eaa2f06bffbcb"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.665908 19311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:28.679735 19311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:28.681946 19311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:28.682142 19311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:28.686446 19311 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:34667
I20260808 21:08:28.686478 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:34667 every 8 connection(s)
I20260808 21:08:28.686818 19311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:28.687801 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:28.688413 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19311
I20260808 21:08:28.688529 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:08:28.688733 4805 external_mini_cluster.cc:1428] Setting key d2bc769439679aaf308b6d4174683d15
I20260808 21:08:28.690085 19383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.125@KRBTEST.COM" created.
I20260808 21:08:28.700594 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.690945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34667 (local address 127.0.0.1:38352)
0808 21:08:28.691221 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.691263 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:28.691752 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:28.692064 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:28.692066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:28.692290 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:28.692434 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.692438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.694028 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.694029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.694688 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.694690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.695463 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.695464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.695526 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.696003 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:28.696010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:28.697355 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:28.698976 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.698982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.698983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.699465 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.699741 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.699741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.699742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.699820 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.700122 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:28.700126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:28.700176 (+ 50us) client_negotiation.cc:769] Sending connection context
0808 21:08:28.700246 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:08:28.700401 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260808 21:08:28.700637 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.691257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.4.177.126:34667)
0808 21:08:28.691399 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:28.691403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:28.691436 (+ 33us) server_negotiation.cc:407] Connection header received
0808 21:08:28.691904 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:28.691906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:28.691928 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:28.692011 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:28.692538 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.693934 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.694778 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.695346 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.695472 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.697440 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:28.697448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:28.697448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:28.697455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:28.698894 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.699539 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.699541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.699542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.699651 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.699878 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.699878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.699879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.700063 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:28.700124 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:28.700328 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 21:08:28.700481 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260808 21:08:28.701347 19383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:08:28.701660 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.701560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39834)
0808 21:08:28.701603 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.701642 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:08:28.701813 19331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
I20260808 21:08:28.702286 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.702165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39844)
0808 21:08:28.702228 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.702259 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260808 21:08:28.702443 19383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34307: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
I20260808 21:08:28.715329 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:34307
--webserver_interface=127.4.177.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:28.751657 19383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } attempt: 1
I20260808 21:08:28.752120 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.751976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39846)
0808 21:08:28.752037 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.752090 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 21:08:28.752556 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.752454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39850)
0808 21:08:28.752500 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.752532 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 21:08:28.752693 19383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34307: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
W20260808 21:08:28.795511 19390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:28.795699 19390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:28.795722 19390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:28.796847 19390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:28.796888 19390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:28.796903 19390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:28.796958 19390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:28.796984 19390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:28.798477 19390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:34307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.177.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:28.798741 19390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:28.798964 19390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:28.803684 19398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:28.803700 19395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:28.803709 19396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:28.803972 19390 server_base.cc:1034] running on GCE node
I20260808 21:08:28.804200 19390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:28.804427 19390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:28.805578 19390 hybrid_clock.cc:648] HybridClock initialized: now 1786223308805563 us; error 32 us; skew 500 ppm
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:28.808179 19390 init.cc:377] Logged in from keytab as kudu/127.4.177.125@KRBTEST.COM (short username kudu)
I20260808 21:08:28.808552 19390 webserver.cc:460] Webserver started at http://127.4.177.125:37733/ using document root <none> and password file <none>
I20260808 21:08:28.808734 19390 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:28.808773 19390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:28.808861 19390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:28.809849 19390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data/instance:
uuid: "fffff087ccfa4202991fca00346145e7"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "8d8e6279a898fe218ff26d5545a8770a"
server_key_iv: "a8e12bbc61f84e9176c71328673a7c4e"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.810148 19390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal/instance:
uuid: "fffff087ccfa4202991fca00346145e7"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "8d8e6279a898fe218ff26d5545a8770a"
server_key_iv: "a8e12bbc61f84e9176c71328673a7c4e"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.811436 19390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:08:28.812142 19405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:28.812325 19390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:28.812417 19390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "fffff087ccfa4202991fca00346145e7"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "8d8e6279a898fe218ff26d5545a8770a"
server_key_iv: "a8e12bbc61f84e9176c71328673a7c4e"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.812490 19390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:28.820834 19383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } attempt: 2
I20260808 21:08:28.821252 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.821108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39852)
0808 21:08:28.821172 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.821215 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 21:08:28.821592 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.821504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39866)
0808 21:08:28.821544 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.821572 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:08:28.821717 19383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34307: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34307: connect: Connection refused (error 111)
I20260808 21:08:28.838961 19390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:28.841441 19390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:28.841624 19390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:28.845860 19390 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.125:34307
I20260808 21:08:28.845929 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.125:34307 every 8 connection(s)
I20260808 21:08:28.846287 19390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
I20260808 21:08:28.847158 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:28.848677 19461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:08:28.849934 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19390
I20260808 21:08:28.850052 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-1/wal/instance
I20260808 21:08:28.850271 4805 external_mini_cluster.cc:1428] Setting key a7a4485382b2d40ba5d8477f6f825d20
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:28.858244 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.849375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.4.177.126:34667)
0808 21:08:28.849425 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:28.849428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:28.849661 (+ 233us) server_negotiation.cc:407] Connection header received
0808 21:08:28.850194 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:28.850196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:28.850212 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:28.850297 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:28.850997 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.851775 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.852819 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.853177 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.853264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.855408 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:28.855409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:28.855410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:28.855417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:28.856841 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.857258 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.857259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.857259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.857325 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.857587 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.857588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.857588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.857709 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:28.857787 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:28.858064 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 21:08:28.858154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:08:28.858438 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.849354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34667 (local address 127.0.0.1:38354)
0808 21:08:28.849536 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.849576 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:28.850077 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:28.850375 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:28.850379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:28.850635 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:28.850894 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.850899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.851865 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.851866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.852723 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.852725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.853276 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.853277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.853377 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.853882 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:28.853899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:28.855311 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:28.856934 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.856943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.856945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.857175 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.857401 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.857402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.857402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.857515 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.857779 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:28.857787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:28.857926 (+ 139us) client_negotiation.cc:769] Sending connection context
0808 21:08:28.858030 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 21:08:28.858196 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.4.177.124@KRBTEST.COM; defaulting to no policy
I20260808 21:08:28.859015 19461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.124@KRBTEST.COM" created.
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:08:28.866940 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.859292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39880)
0808 21:08:28.859346 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.859363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:28.859395 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:28.859749 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:28.859750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:28.859881 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:28.859929 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.859931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.860860 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.860861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.861475 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.861477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.862030 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.862031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.862100 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.862526 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:28.862531 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:28.863889 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:28.865944 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.865945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.865945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.866104 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.866349 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.866349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.866350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.866399 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.866728 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:28.866729 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:28.866763 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 21:08:28.866810 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:08:28.866877 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 21:08:28.867118 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.859369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.4.177.125:34307)
0808 21:08:28.859590 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:28.859595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:28.859618 (+ 23us) server_negotiation.cc:407] Connection header received
0808 21:08:28.859637 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:28.859639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:28.859652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:28.859735 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:28.860006 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.860753 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.861561 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.861929 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.862032 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.863968 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:28.863976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:28.863977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:28.863984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:28.865862 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.866200 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.866204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.866205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.866275 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.866458 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.866459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.866460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.866653 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:28.866741 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:28.866850 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 21:08:28.866967 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260808 21:08:28.867832 19461 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:08:28.868166 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.868062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57110)
0808 21:08:28.868098 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.868148 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260808 21:08:28.868306 19406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
I20260808 21:08:28.868736 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.868659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57120)
0808 21:08:28.868689 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.868713 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260808 21:08:28.868839 19461 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39305: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
I20260808 21:08:28.878149 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:39305
--webserver_interface=127.4.177.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:28.912850 19383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } attempt: 3
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:08:28.918080 19461 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } attempt: 1
I20260808 21:08:28.918494 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.918379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57134)
0808 21:08:28.918428 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.918456 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 21:08:28.919553 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.919417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57144)
0808 21:08:28.919483 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.919516 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 21:08:28.919731 19461 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39305: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
I20260808 21:08:28.920396 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.913342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.4.177.125:34307)
0808 21:08:28.913418 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:28.913420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:28.913430 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:08:28.913461 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:28.913462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:28.913474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:28.913558 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:28.913892 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.914430 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.915252 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.915465 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.915554 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.917522 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:28.917524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:28.917524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:28.917532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:28.919127 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.919561 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.919563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.919563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.919631 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:28.919864 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:28.919865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:28.919865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:28.919984 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:28.920066 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:28.920237 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 21:08:28.920310 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 21:08:28.920403 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.913246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39886)
0808 21:08:28.913293 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.913306 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:28.913400 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:28.913572 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:28.913573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:28.913737 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:28.913803 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.913805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.914530 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.914532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.915119 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:28.915122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:28.915555 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:28.915556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:28.915650 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:28.916175 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:28.916187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:28.917426 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:28.919250 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.919251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.919251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.919480 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.919709 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:28.919709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:28.919710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:28.919786 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:28.920060 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:28.920060 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:28.920118 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 21:08:28.920199 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 21:08:28.920306 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260808 21:08:28.920953 19383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:08:28.921271 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.921191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57150)
0808 21:08:28.921222 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.921250 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 21:08:28.921648 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.921553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57156)
0808 21:08:28.921600 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.921624 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 21:08:28.921734 19383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39305: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
I20260808 21:08:28.952870 19383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } attempt: 1
I20260808 21:08:28.953403 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.953263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57168)
0808 21:08:28.953328 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.953375 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 21:08:28.953794 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.953655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57182)
0808 21:08:28.953723 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.953765 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 21:08:28.953931 19383 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39305: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
W20260808 21:08:28.961999 19468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:28.962198 19468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:28.962221 19468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:28.963420 19468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:28.963459 19468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:28.963474 19468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:28.963505 19468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:28.963531 19468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:28.965006 19468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:34667,127.4.177.125:34307,127.4.177.124:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.177.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:28.965252 19468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:28.965481 19468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:28.970355 19473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:28.970407 19474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:28.970345 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:28.970639 19468 server_base.cc:1034] running on GCE node
I20260808 21:08:28.970956 19468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:28.971294 19468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:28.972470 19468 hybrid_clock.cc:648] HybridClock initialized: now 1786223308972459 us; error 24 us; skew 500 ppm
Aug 08 21:08:28 dist-test-slave-chsk krb5kdc[19296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:28.974854 19468 init.cc:377] Logged in from keytab as kudu/127.4.177.124@KRBTEST.COM (short username kudu)
I20260808 21:08:28.975378 19468 webserver.cc:460] Webserver started at http://127.4.177.124:44509/ using document root <none> and password file <none>
I20260808 21:08:28.975625 19468 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:28.975696 19468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:28.975848 19468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:28.976847 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data/instance:
uuid: "27b55c3b18294a1e84c6e1e367d671af"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "3d622468fabf98b8b703a7a2152996ed"
server_key_iv: "58e23c6c667b87b7ab961ae40099a71c"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.977209 19468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal/instance:
uuid: "27b55c3b18294a1e84c6e1e367d671af"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "3d622468fabf98b8b703a7a2152996ed"
server_key_iv: "58e23c6c667b87b7ab961ae40099a71c"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.978490 19468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:08:28.979230 19484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:28.979405 19468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:28.979501 19468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "27b55c3b18294a1e84c6e1e367d671af"
format_stamp: "Formatted at 2026-08-08 21:08:28 on dist-test-slave-chsk"
server_key: "3d622468fabf98b8b703a7a2152996ed"
server_key_iv: "58e23c6c667b87b7ab961ae40099a71c"
server_key_version: "encryptionkey@0"
I20260808 21:08:28.979588 19468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:28.987875 19461 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } attempt: 2
I20260808 21:08:28.988286 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.988160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57194)
0808 21:08:28.988221 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.988257 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:08:28.988716 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:28.988611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57200)
0808 21:08:28.988658 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:28.988684 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:08:28.988809 19461 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39305: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39305: connect: Connection refused (error 111)
I20260808 21:08:28.995630 19468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:28.997371 19468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:28.997529 19468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:29.001842 19468 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.124:39305
I20260808 21:08:29.001875 19539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.124:39305 every 8 connection(s)
I20260808 21:08:29.002228 19468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
I20260808 21:08:29.003125 19540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:29.003237 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19468
I20260808 21:08:29.003357 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786223155551886-4805-0/minicluster-data/master-2/wal/instance
I20260808 21:08:29.003584 4805 external_mini_cluster.cc:1428] Setting key 17480e42d095b2929d298d883f03bcc7
I20260808 21:08:29.005683 19540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:29.014909 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.004124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.4.177.124:39305)
0808 21:08:29.004312 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.004320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.004344 (+ 24us) server_negotiation.cc:407] Connection header received
0808 21:08:29.004765 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.004771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.004815 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.004923 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.005728 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.006808 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.007719 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.008258 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.008340 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.011324 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.011332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.011333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.011340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.013355 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.013706 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.013710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.013710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.013813 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.014005 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.014006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.014006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.014252 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.014411 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.014519 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.014666 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260808 21:08:29.015142 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.006307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:34667 (local address 127.0.0.1:38360)
0808 21:08:29.006460 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:29.006505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:29.006594 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:29.006787 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:29.006790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:29.007190 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:29.007281 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.007284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.007999 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.008000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.008770 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.008773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.009153 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.009154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.009233 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.009635 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:29.009645 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:29.011773 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:29.014052 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.014058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.014059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.014231 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.014448 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.014448 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.014449 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.014515 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.014806 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:29.014812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:29.014852 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 21:08:29.014916 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 21:08:29.015030 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260808 21:08:29.015192 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.006393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.4.177.126:34667)
0808 21:08:29.006447 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.006449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.006576 (+ 127us) server_negotiation.cc:407] Connection header received
0808 21:08:29.006674 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.006676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.006687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.006764 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.007371 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.007914 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.008851 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.009058 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.009141 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.011867 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.011873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.011873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.011879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.013982 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.014303 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.014304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.014305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.014366 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.014601 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.014602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.014602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.014734 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.014807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.014949 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.015038 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:08:29.015837 19540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:29.023591 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.016082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34307 (local address 127.0.0.1:39900)
0808 21:08:29.016127 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:29.016137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:29.016177 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:29.016364 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:29.016364 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:29.016471 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:29.016497 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.016498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.017333 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.017333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.018072 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.018074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.018512 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.018513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.018569 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.019007 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:29.019012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:29.020281 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:29.022552 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.022553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.022553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.022710 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.022912 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.022912 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.022913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.022959 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.023302 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:29.023303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:29.023346 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 21:08:29.023420 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 21:08:29.023498 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 21:08:29.023638 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.016174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.4.177.125:34307)
0808 21:08:29.016244 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.016247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.016256 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:29.016282 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.016284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.016294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.016361 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.016556 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.017262 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.018146 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.018433 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.018500 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.020366 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.020367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.020368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.020376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.022465 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.022787 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.022788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.022788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.022844 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.023052 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.023053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.023053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.023233 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.023296 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.023458 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.023539 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:08:29.023959 19540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:08:29.024108 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.015738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.4.177.126:34667)
0808 21:08:29.015776 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.015777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.015787 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:08:29.015841 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.015842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.015851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.015905 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.016476 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.017173 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.018140 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.018432 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.018500 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.021022 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.021023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.021024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.021031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.022709 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.023259 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.023260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.023260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.023339 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.023579 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.023579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.023580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.023713 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.023886 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.023923 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.024001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:08:29.029155 19383 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } attempt: 2
I20260808 21:08:29.031169 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.024307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57220)
0808 21:08:29.024345 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:29.024357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:29.024398 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:29.024579 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:29.024580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:29.024700 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:29.024746 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.024747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.025435 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.025436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.025930 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.025931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.026264 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.026265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.026316 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.026723 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:29.026728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:29.028084 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:29.030122 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.030123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.030123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.030308 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.030522 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.030523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.030523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.030579 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.030824 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:29.030825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:29.030860 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 21:08:29.030928 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:08:29.031025 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260808 21:08:29.031512 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.024372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.4.177.124:39305)
0808 21:08:29.024472 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.024474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.024481 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:08:29.024499 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.024499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.024507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.024561 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.024812 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.025355 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.026008 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.026196 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.026246 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.028170 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.028171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.028171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.028178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.030056 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.030399 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.030400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.030400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.030454 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.030653 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.030654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.030654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.030760 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.030832 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.030954 (+ 122us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.031426 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":393,"server-negotiator.queue_time_us":80}
I20260808 21:08:29.032508 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.024747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.4.177.125:34307)
0808 21:08:29.024780 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.024782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.024791 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:29.024848 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.024849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.024857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.024911 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.025314 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.025789 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.026599 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.026795 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.026866 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.029076 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.029078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.029078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.029085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.030928 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.031372 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.031373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.031374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.031439 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.031650 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.031652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.031652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.031769 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.032345 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.032389 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.032450 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 21:08:29.033272 19540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: Bootstrap starting.
I20260808 21:08:29.033931 19540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: Neither blocks nor log segments found. Creating new log.
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:08:29.034298 19540 log.cc:824] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:29.034337 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:08:29.035856 19540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: No bootstrap required, opened a new log
I20260808 21:08:29.037591 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.029424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57226)
0808 21:08:29.029468 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:29.029477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:29.029512 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:29.029777 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:29.029778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:29.029918 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:29.029977 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.029978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.031415 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.031416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.032171 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.032174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.032650 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.032651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.032709 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.033101 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:29.033106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:29.034465 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:29.036438 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.036439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.036440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.036662 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.036892 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.036893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.036893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.036956 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.037275 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:29.037276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:29.037320 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 21:08:29.037387 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 21:08:29.037482 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 21:08:29.038152 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.029503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.4.177.124:39305)
0808 21:08:29.029664 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.029666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.029672 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:08:29.029686 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.029687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.029694 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.029772 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.030101 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.031317 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.032275 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.032539 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.032637 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.034549 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.034550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.034551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.034558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.036362 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.036750 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.036751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.036751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.036821 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.037021 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.037022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.037022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.037181 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.037244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.037967 (+ 723us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.038049 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260808 21:08:29.038170 19540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.038316 19540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:29.038359 19540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b55c3b18294a1e84c6e1e367d671af, State: Initialized, Role: FOLLOWER
I20260808 21:08:29.038489 19540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.038822 19540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:29.039208 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.039397 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.039824 19383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: Bootstrap starting.
I20260808 21:08:29.040534 19383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:29.040846 19383 log.cc:824] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: Log is configured to *not* fsync() on all Append() calls
W20260808 21:08:29.041993 19568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:08:29.042114 19568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:08:29.042976 19383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: No bootstrap required, opened a new log
I20260808 21:08:29.044032 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.035937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.4.177.126:34667)
0808 21:08:29.035976 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.035978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.036694 (+ 716us) server_negotiation.cc:407] Connection header received
0808 21:08:29.036791 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.036792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.036802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.036833 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.037512 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.038216 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.039089 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.039427 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.039500 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.040931 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.040932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.040933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.040941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.042576 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.042971 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.042972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.042973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.043068 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.043366 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.043367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.043368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.043491 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.043575 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.043879 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.043949 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:08:29.045349 19383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.045483 19383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:29.042512 19468 master_runner.cc:186] Error getting master registration for 127.4.177.126:34667: OK, instance_id { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" instance_seqno: 1786223308685195 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:08:29.045524 19383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a64f3cf06ec4d92bf6afda23364f1c0, State: Initialized, Role: FOLLOWER
I20260808 21:08:29.045610 19468 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:08:29.045655 19383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.046000 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.035700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.4.177.124:39305)
0808 21:08:29.035741 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.035743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.035885 (+ 142us) server_negotiation.cc:407] Connection header received
0808 21:08:29.035981 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.035982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.035991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.036020 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.036795 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.037507 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.041147 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.041453 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.041497 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.043117 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.043118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.043118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.043125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.044781 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.045176 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.045177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.045177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.045231 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.045436 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.045437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.045437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.045559 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.045644 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.045843 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.045918 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:08:29.046162 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.046276 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.046665 19383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 21:08:29.048795 19584 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:08:29.048911 19584 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:08:29.051352 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.036240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.4.177.125:34307)
0808 21:08:29.036274 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.036276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.041989 (+ 5713us) server_negotiation.cc:407] Connection header received
0808 21:08:29.042136 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.042137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.042149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.042183 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.044417 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.045105 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.046058 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.046391 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.046451 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.047952 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.047953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.047954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.047962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.049732 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.050174 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.050175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.050175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.050250 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.050498 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.050499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.050500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.050646 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.050736 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.051183 (+ 447us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.051264 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:08:29.049707 19311 master_runner.cc:186] Error getting master registration for 127.4.177.125:34307: OK, instance_id { permanent_uuid: "fffff087ccfa4202991fca00346145e7" instance_seqno: 1786223308844669 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:08:29.052067 19311 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:08:29.079957 19461 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } attempt: 3
Aug 08 21:08:29 dist-test-slave-chsk krb5kdc[19296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:08:29.088842 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.080277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39305 (local address 127.0.0.1:57238)
0808 21:08:29.080335 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:29.080346 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:29.080384 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:29.080527 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:29.080528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:29.080735 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:29.080793 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.080795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.081760 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.081761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.082625 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:29.082628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.083121 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.083122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:29.083219 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.083750 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:29.083756 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:29.085329 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:29.087584 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.087586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.087587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.087809 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.088027 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:29.088028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:29.088028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:29.088096 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:29.088447 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:29.088448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:29.088497 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 21:08:29.088576 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:08:29.088709 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 21:08:29.088855 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:29.080340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.4.177.124:39305)
0808 21:08:29.080389 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:29.080392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:29.080403 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:08:29.080444 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:29.080445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:29.080455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:29.080509 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:29.080873 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.081678 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.082743 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:29.083044 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:29.083110 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:29.085421 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:29.085422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:29.085422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:29.085430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:29.087493 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.087891 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.087892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.087892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.087954 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:29.088163 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:29.088164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:29.088165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:29.088373 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:29.088444 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:29.088631 (+ 187us) server_negotiation.cc:299] Negotiation successful
0808 21:08:29.088759 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:08:29.090711 19461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: Bootstrap starting.
I20260808 21:08:29.091497 19461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:29.091822 19461 log.cc:824] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:29.092780 19461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: No bootstrap required, opened a new log
I20260808 21:08:29.095111 19461 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.095283 19461 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:29.095319 19461 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffff087ccfa4202991fca00346145e7, State: Initialized, Role: FOLLOWER
I20260808 21:08:29.095480 19461 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.095808 19461 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 21:08:29.097725 19604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:08:29.097833 19604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:08:29.099718 19390 master_runner.cc:263] No leader master found from master fffff087ccfa4202991fca00346145e7
I20260808 21:08:29.099768 19390 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:08:29.095791 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.099936 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.119302 19556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:29.119421 19556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.119668 19556 leader_election.cc:288] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a64f3cf06ec4d92bf6afda23364f1c0 (127.4.177.126:34667), fffff087ccfa4202991fca00346145e7 (127.4.177.125:34307)
I20260808 21:08:29.120119 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27b55c3b18294a1e84c6e1e367d671af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" is_pre_election: true
I20260808 21:08:29.120265 19357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b55c3b18294a1e84c6e1e367d671af in term 0.
I20260808 21:08:29.120568 19433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27b55c3b18294a1e84c6e1e367d671af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffff087ccfa4202991fca00346145e7" is_pre_election: true
I20260808 21:08:29.120709 19433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b55c3b18294a1e84c6e1e367d671af in term 0.
I20260808 21:08:29.121675 19490 leader_election.cc:302] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b55c3b18294a1e84c6e1e367d671af, fffff087ccfa4202991fca00346145e7; no voters:
I20260808 21:08:29.122102 19556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:08:29.122169 19556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:29.122195 19556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:29.123044 19556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.123160 19556 leader_election.cc:288] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [CANDIDATE]: Term 1 election: Requested vote from peers 1a64f3cf06ec4d92bf6afda23364f1c0 (127.4.177.126:34667), fffff087ccfa4202991fca00346145e7 (127.4.177.125:34307)
I20260808 21:08:29.127362 19357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27b55c3b18294a1e84c6e1e367d671af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0"
I20260808 21:08:29.127446 19357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:29.128332 19357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b55c3b18294a1e84c6e1e367d671af in term 1.
I20260808 21:08:29.128571 19433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "27b55c3b18294a1e84c6e1e367d671af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fffff087ccfa4202991fca00346145e7"
I20260808 21:08:29.128652 19433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:29.129510 19433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b55c3b18294a1e84c6e1e367d671af in term 1.
I20260808 21:08:29.129745 19490 leader_election.cc:302] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b55c3b18294a1e84c6e1e367d671af, fffff087ccfa4202991fca00346145e7; no voters:
I20260808 21:08:29.129895 19556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:29.130066 19556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [term 1 LEADER]: Becoming Leader. State: Replica: 27b55c3b18294a1e84c6e1e367d671af, State: Running, Role: LEADER
I20260808 21:08:29.130155 19556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } }
I20260808 21:08:29.130527 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b55c3b18294a1e84c6e1e367d671af. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.130628 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:29.135747 19609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:29.136008 19609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:29.141719 19433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [term 1 FOLLOWER]: Refusing update from remote peer 27b55c3b18294a1e84c6e1e367d671af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:29.142158 19556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [LEADER]: Connected to new peer: Peer: permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:29.143635 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b55c3b18294a1e84c6e1e367d671af. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.143724 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.144170 19357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [term 1 FOLLOWER]: Refusing update from remote peer 27b55c3b18294a1e84c6e1e367d671af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:29.144479 19556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:29.145722 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27b55c3b18294a1e84c6e1e367d671af. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.145843 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.147311 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.147400 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:29.148000 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.148085 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.148277 19610 mvcc.cc:205] Tried to move back new op lower bound from 7316370674239942656 to 7316370674197729280. Current Snapshot: MvccSnapshot[applied={T|T < 7316370674239942656}]
I20260808 21:08:29.148422 19611 mvcc.cc:205] Tried to move back new op lower bound from 7316370674239942656 to 7316370674197729280. Current Snapshot: MvccSnapshot[applied={T|T < 7316370674239942656}]
I20260808 21:08:29.149312 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.149390 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:29.149854 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27b55c3b18294a1e84c6e1e367d671af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a64f3cf06ec4d92bf6afda23364f1c0" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34667 } } peers { permanent_uuid: "fffff087ccfa4202991fca00346145e7" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34307 } } peers { permanent_uuid: "27b55c3b18294a1e84c6e1e367d671af" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39305 } } }
I20260808 21:08:29.149935 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:08:29.150218 19613 mvcc.cc:205] Tried to move back new op lower bound from 7316370674239942656 to 7316370674197729280. Current Snapshot: MvccSnapshot[applied={T|T < 7316370674239942656}]
I20260808 21:08:29.152355 19609 catalog_manager.cc:1347] Generated new cluster ID: 2ab4fa3673914fbe9c9e70b4fa971a01
I20260808 21:08:29.152415 19609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:08:29.176465 19609 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:08:29.177386 19609 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:29.199105 19609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27b55c3b18294a1e84c6e1e367d671af: Generated new TSK 0
I20260808 21:08:29.199301 19609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:08:30.049396 19584 catalog_manager.cc:1259] Loaded cluster ID: 2ab4fa3673914fbe9c9e70b4fa971a01
I20260808 21:08:30.049482 19584 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: loading cluster ID for follower catalog manager: success
I20260808 21:08:30.050290 19584 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: acquiring CA information for follower catalog manager: success
I20260808 21:08:30.050637 19584 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1a64f3cf06ec4d92bf6afda23364f1c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:08:30.098292 19604 catalog_manager.cc:1259] Loaded cluster ID: 2ab4fa3673914fbe9c9e70b4fa971a01
I20260808 21:08:30.098372 19604 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: loading cluster ID for follower catalog manager: success
I20260808 21:08:30.099174 19604 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: acquiring CA information for follower catalog manager: success
I20260808 21:08:30.099431 19604 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fffff087ccfa4202991fca00346145e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:08:30.260752 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19311
I20260808 21:08:30.268433 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19390
I20260808 21:08:30.275341 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19468
2026-08-08T21:08:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2946 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:08:30 dist-test-slave-chsk krb5kdc[19624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:08:30 dist-test-slave-chsk krb5kdc[19624](info): set up 2 sockets
Aug 08 21:08:30 dist-test-slave-chsk krb5kdc[19624](info): commencing operation
krb5kdc: starting...
W20260808 21:08:31.351106 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:08:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:08:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:08:31.473974 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41843
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:31.556403 19637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:31.556670 19637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:31.556711 19637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:31.557905 19637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:31.557976 19637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:31.558000 19637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:31.558022 19637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:31.558051 19637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:31.559614 19637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:31.559906 19637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:31.560151 19637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:31.565061 19648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:31.565073 19645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:31.565141 19644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:31.565477 19637 server_base.cc:1034] running on GCE node
I20260808 21:08:31.565723 19637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:31.566025 19637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:31.567242 19637 hybrid_clock.cc:648] HybridClock initialized: now 1786223311567223 us; error 29 us; skew 500 ppm
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:31.569816 19637 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:31.570319 19637 webserver.cc:460] Webserver started at http://127.4.177.126:42325/ using document root <none> and password file <none>
I20260808 21:08:31.570564 19637 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:31.570636 19637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:31.570791 19637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:31.572094 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "9427404e0e074a5c96d57e10dd3b491a"
format_stamp: "Formatted at 2026-08-08 21:08:31 on dist-test-slave-chsk"
server_key: "da2202a368fbac40a537d3e3a5fe63af"
server_key_iv: "583af00f0e9f291a17debf7359b23070"
server_key_version: "encryptionkey@0"
I20260808 21:08:31.572479 19637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "9427404e0e074a5c96d57e10dd3b491a"
format_stamp: "Formatted at 2026-08-08 21:08:31 on dist-test-slave-chsk"
server_key: "da2202a368fbac40a537d3e3a5fe63af"
server_key_iv: "583af00f0e9f291a17debf7359b23070"
server_key_version: "encryptionkey@0"
I20260808 21:08:31.573838 19637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:08:31.574616 19655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:31.574802 19637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:08:31.574909 19637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "9427404e0e074a5c96d57e10dd3b491a"
format_stamp: "Formatted at 2026-08-08 21:08:31 on dist-test-slave-chsk"
server_key: "da2202a368fbac40a537d3e3a5fe63af"
server_key_iv: "583af00f0e9f291a17debf7359b23070"
server_key_version: "encryptionkey@0"
I20260808 21:08:31.574985 19637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:31.586874 19637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:31.589228 19637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:31.589401 19637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:31.593676 19637 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:41843
I20260808 21:08:31.593715 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:41843 every 8 connection(s)
I20260808 21:08:31.594112 19637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:31.595058 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:31.597386 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a: Bootstrap starting.
I20260808 21:08:31.598001 19711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:31.598259 19711 log.cc:824] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:31.598907 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19637
I20260808 21:08:31.599022 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:08:31.599036 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a: No bootstrap required, opened a new log
I20260808 21:08:31.599277 4805 external_mini_cluster.cc:1428] Setting key f008288942d1866a8f1df9c98fd44985
I20260808 21:08:31.601281 19711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } }
I20260808 21:08:31.601459 19711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:31.601500 19711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9427404e0e074a5c96d57e10dd3b491a, State: Initialized, Role: FOLLOWER
I20260808 21:08:31.601662 19711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } }
I20260808 21:08:31.601747 19711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:31.601770 19711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:31.601841 19711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:31.603015 19711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } }
I20260808 21:08:31.603124 19711 leader_election.cc:302] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9427404e0e074a5c96d57e10dd3b491a; no voters:
I20260808 21:08:31.603303 19711 leader_election.cc:288] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:08:31.603369 19716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:31.603518 19716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [term 1 LEADER]: Becoming Leader. State: Replica: 9427404e0e074a5c96d57e10dd3b491a, State: Running, Role: LEADER
I20260808 21:08:31.603539 19711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:31.603606 19716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } }
I20260808 21:08:31.604046 19717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9427404e0e074a5c96d57e10dd3b491a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } } }
I20260808 21:08:31.604135 19717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:31.604314 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9427404e0e074a5c96d57e10dd3b491a. Latest consensus state: current_term: 1 leader_uuid: "9427404e0e074a5c96d57e10dd3b491a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9427404e0e074a5c96d57e10dd3b491a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41843 } } }
I20260808 21:08:31.604368 19727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:31.604384 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:31.605046 19727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:31.607084 19727 catalog_manager.cc:1347] Generated new cluster ID: cef63a179abf48e2b5eff235acfe908f
I20260808 21:08:31.607133 19727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:31.612404 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:31.599856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.4.177.126:41843)
0808 21:08:31.600076 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:31.600086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:31.600123 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:08:31.601243 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:31.601253 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:31.601319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:31.601477 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:31.602314 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:31.603776 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:31.604967 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:31.605590 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:31.605727 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:31.608682 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:31.608686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:31.608687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:31.608698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:31.610671 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:31.611098 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:31.611101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:31.611101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:31.611283 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:31.611526 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:31.611526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:31.611527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:31.611811 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:31.611955 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:31.612073 (+ 118us) server_negotiation.cc:299] Negotiation successful
0808 21:08:31.612202 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260808 21:08:31.613317 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:08:31.622541 19727 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:08:31.623092 19727 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:31.628654 19727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9427404e0e074a5c96d57e10dd3b491a: Generated new TSK 0
I20260808 21:08:31.628851 19727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:08:31.667351 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19637
2026-08-08T21:08:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1390 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19740](info): set up 2 sockets
Aug 08 21:08:31 dist-test-slave-chsk krb5kdc[19740](info): commencing operation
krb5kdc: starting...
W20260808 21:08:32.747849 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:08:32 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:08:32.884009 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:32.967778 19755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:32.967991 19755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:32.968014 19755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:32.969139 19755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:32.969223 19755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:32.969259 19755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:32.969287 19755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:32.969300 19755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:32.970750 19755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:32.970947 19755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:32.971118 19755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:32.976047 19761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:32.976014 19760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:32.976030 19763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:32.976460 19755 server_base.cc:1034] running on GCE node
I20260808 21:08:32.976738 19755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:32.977026 19755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:32.978180 19755 hybrid_clock.cc:648] HybridClock initialized: now 1786223312978166 us; error 34 us; skew 500 ppm
Aug 08 21:08:32 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223312, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:32.980909 19755 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:32.981352 19755 webserver.cc:460] Webserver started at http://127.4.177.126:46549/ using document root <none> and password file <none>
I20260808 21:08:32.981577 19755 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:32.981622 19755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:32.981791 19755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:32.982862 19755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b71e8a690517484ba3e5aae7e916020e"
format_stamp: "Formatted at 2026-08-08 21:08:32 on dist-test-slave-chsk"
server_key: "0f9474c282405f51b2c9818b26805887"
server_key_iv: "7f35880aae7e37e2c16ad6cfb7a998b4"
server_key_version: "encryptionkey@0"
I20260808 21:08:32.983265 19755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b71e8a690517484ba3e5aae7e916020e"
format_stamp: "Formatted at 2026-08-08 21:08:32 on dist-test-slave-chsk"
server_key: "0f9474c282405f51b2c9818b26805887"
server_key_iv: "7f35880aae7e37e2c16ad6cfb7a998b4"
server_key_version: "encryptionkey@0"
I20260808 21:08:32.984634 19755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:32.985370 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:32.985564 19755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:32.985661 19755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b71e8a690517484ba3e5aae7e916020e"
format_stamp: "Formatted at 2026-08-08 21:08:32 on dist-test-slave-chsk"
server_key: "0f9474c282405f51b2c9818b26805887"
server_key_iv: "7f35880aae7e37e2c16ad6cfb7a998b4"
server_key_version: "encryptionkey@0"
I20260808 21:08:32.985741 19755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:33.003244 19755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:33.005689 19755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:33.005867 19755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:33.010206 19755 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:37635
I20260808 21:08:33.010267 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:37635 every 8 connection(s)
I20260808 21:08:33.010622 19755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:33.011698 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:33.013959 19828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap starting.
I20260808 21:08:33.014499 19828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:33.014758 19828 log.cc:824] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:33.015509 19828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: No bootstrap required, opened a new log
I20260808 21:08:33.017112 19828 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.017253 19828 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:33.017277 19828 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Initialized, Role: FOLLOWER
I20260808 21:08:33.017382 19828 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.017432 19828 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:33.017453 19828 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:33.017481 19828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:33.018267 19828 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.018356 19828 leader_election.cc:302] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71e8a690517484ba3e5aae7e916020e; no voters:
I20260808 21:08:33.018481 19828 leader_election.cc:288] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:08:33.018534 19831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:33.018685 19831 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 LEADER]: Becoming Leader. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Running, Role: LEADER
I20260808 21:08:33.018783 19828 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:33.018934 19831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.019361 19833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71e8a690517484ba3e5aae7e916020e. Latest consensus state: current_term: 1 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:33.019352 19832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:33.019508 19833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:33.019517 19832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:33.019778 19839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:33.020079 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19755
I20260808 21:08:33.020215 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:08:33.020350 19839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:33.020452 4805 external_mini_cluster.cc:1428] Setting key 25be5ee8a86a757b98e3aba10caa72ad
I20260808 21:08:33.021997 19839 catalog_manager.cc:1347] Generated new cluster ID: f28a695b80444e2baf5d189d6a460229
I20260808 21:08:33.022076 19839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:33.028949 19839 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:08:33.029608 19839 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:33.032511 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.021452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.4.177.126:37635)
0808 21:08:33.021611 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.021620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.023388 (+ 1768us) server_negotiation.cc:407] Connection header received
0808 21:08:33.023548 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.023554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.023598 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.023727 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:33.024368 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.025276 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.026012 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.026423 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.026551 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.029048 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:33.029051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:33.029052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:33.029063 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:33.030881 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.031330 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.031333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.031334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.031431 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.031647 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.031648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.031648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.031884 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:33.032062 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.032141 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.032303 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260808 21:08:33.035718 19839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Generated new TSK 0
I20260808 21:08:33.035903 19839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:08:33.063287 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:33.148447 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:33.148680 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:33.148721 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:33.149925 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:33.149996 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:33.150041 19855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:33.151567 19855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:33.151855 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:33.152099 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:33.157984 19862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.157992 19864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.158185 19855 server_base.cc:1034] running on GCE node
W20260808 21:08:33.158057 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.158483 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:33.158769 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:33.159910 19855 hybrid_clock.cc:648] HybridClock initialized: now 1786223313159900 us; error 30 us; skew 500 ppm
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:33.162565 19855 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:33.163055 19855 webserver.cc:460] Webserver started at http://127.4.177.65:34145/ using document root <none> and password file <none>
I20260808 21:08:33.163347 19855 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:33.163417 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:33.163563 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:33.164609 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "8a523e37cfd147828101c003af349c1c"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "cf9d108de3cf812105c60707077d2da9"
server_key_iv: "64bfb896ab54c7d30202ee6a2136de24"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.164983 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "8a523e37cfd147828101c003af349c1c"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "cf9d108de3cf812105c60707077d2da9"
server_key_iv: "64bfb896ab54c7d30202ee6a2136de24"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.166320 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:08:33.167109 19871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.167336 19855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:33.167434 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "8a523e37cfd147828101c003af349c1c"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "cf9d108de3cf812105c60707077d2da9"
server_key_iv: "64bfb896ab54c7d30202ee6a2136de24"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.167491 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:33.178839 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:33.180801 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:33.180915 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:33.181102 19855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:33.181442 19855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:33.181478 19855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.181500 19855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:33.181550 19855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.187314 19855 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35989
I20260808 21:08:33.187409 19992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35989 every 8 connection(s)
I20260808 21:08:33.187722 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:08:33.188941 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19855
I20260808 21:08:33.189045 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:08:33.189277 4805 external_mini_cluster.cc:1428] Setting key e5b73aa7c9e5ab0b2fec2d2d2d570783
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
I20260808 21:08:33.198834 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.188476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:51537 (local address 127.4.177.126:37635)
0808 21:08:33.188536 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.188538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.188785 (+ 247us) server_negotiation.cc:407] Connection header received
0808 21:08:33.189564 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.189565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.189586 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.189646 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:33.190740 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.191565 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.193006 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.193232 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.193295 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.195843 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:33.195845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:33.195845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:33.195853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:33.197310 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.197750 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.197752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.197752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.197820 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.198094 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.198095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.198095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.198213 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:33.198289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.198602 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.198722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:08:33.198870 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.188465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:51537)
0808 21:08:33.188642 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.188677 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.189402 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.189739 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.189743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.190440 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:33.190629 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.190633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.191656 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.191657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.192899 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.192902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.193327 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.193329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.193477 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.194012 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:33.194022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:33.195743 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:33.197416 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.197425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.197427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.197669 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.197891 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.197892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.197893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.198016 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.198280 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:33.198289 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:33.198433 (+ 144us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.198555 (+ 122us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.198700 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260808 21:08:33.199486 19993 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:33.199595 19993 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:33.199786 19993 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:33.200351 19790 ts_manager.cc:194] Registered new tserver with Master: 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989)
I20260808 21:08:33.201011 19790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:51537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:08:33.216020 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:33.299351 19999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:33.299551 19999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:33.299574 19999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:33.300779 19999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:33.300825 19999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:33.300851 19999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:33.302322 19999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:33.302567 19999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:33.302825 19999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:08:33.307919 19999 server_base.cc:1034] running on GCE node
W20260808 21:08:33.307896 20005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.307889 20004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.307896 20007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.308338 19999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:33.308624 19999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:33.309798 19999 hybrid_clock.cc:648] HybridClock initialized: now 1786223313309780 us; error 44 us; skew 500 ppm
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:33.312556 19999 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:33.312999 19999 webserver.cc:460] Webserver started at http://127.4.177.66:40765/ using document root <none> and password file <none>
I20260808 21:08:33.313252 19999 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:33.313323 19999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:33.313477 19999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:33.314534 19999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "386b2a0f586f4a819f071d48b2cdd234"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "18ab915c0767e65ac0d53a5d8589a835"
server_key_iv: "efdbecadcde0997f3c5a711f6c29a7ed"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.314911 19999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "386b2a0f586f4a819f071d48b2cdd234"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "18ab915c0767e65ac0d53a5d8589a835"
server_key_iv: "efdbecadcde0997f3c5a711f6c29a7ed"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.316411 19999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:08:33.317206 20015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.317389 19999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:33.317483 19999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "386b2a0f586f4a819f071d48b2cdd234"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "18ab915c0767e65ac0d53a5d8589a835"
server_key_iv: "efdbecadcde0997f3c5a711f6c29a7ed"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.317576 19999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:33.335412 19999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:33.337867 19999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:33.338045 19999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:33.338306 19999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:33.338646 19999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:33.338703 19999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.338740 19999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:33.338788 19999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.344672 19999 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38427
I20260808 21:08:33.344719 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38427 every 8 connection(s)
I20260808 21:08:33.345086 19999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:33.351758 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 19999
I20260808 21:08:33.351892 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:08:33.352118 4805 external_mini_cluster.cc:1428] Setting key 3281bb762d4dcc70eaff1077afa3821f
I20260808 21:08:33.354635 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.345836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:37773)
0808 21:08:33.345998 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.346041 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.346522 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.346751 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.346754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.347414 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:33.347602 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.347606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.348245 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.348246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.349022 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.349023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.349352 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.349353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.349451 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.349924 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:33.349932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:33.351366 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:33.353272 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.353278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.353280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.353548 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.353779 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.353780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.353780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.353879 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.354159 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:33.354163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:33.354277 (+ 114us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.354360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.354476 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260808 21:08:33.354643 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.345876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:37773 (local address 127.4.177.126:37635)
0808 21:08:33.345924 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.345926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.346133 (+ 207us) server_negotiation.cc:407] Connection header received
0808 21:08:33.346606 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.346607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.346617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.346678 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:33.347684 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.348168 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.349097 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.349276 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.349331 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.351461 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:33.351463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:33.351463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:33.351471 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:33.353192 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.353639 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.353640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.353640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.353708 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.353945 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.353945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.353946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.354094 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:33.354165 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.354405 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.354494 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:08:33.355245 20139 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:33.355338 20139 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:33.355521 20139 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:33.355934 19790 ts_manager.cc:194] Registered new tserver with Master: 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427)
I20260808 21:08:33.356309 19790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:37773
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:08:33.380601 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:33.462894 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:33.463091 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:33.463114 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:33.464278 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:33.464321 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:33.464346 20144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:33.465859 20144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:33.466095 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:33.466322 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:33.471205 20149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.471149 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.471259 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.471398 20144 server_base.cc:1034] running on GCE node
I20260808 21:08:33.471655 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:33.471928 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:33.473084 20144 hybrid_clock.cc:648] HybridClock initialized: now 1786223313473071 us; error 25 us; skew 500 ppm
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:33.475745 20144 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:33.476300 20144 webserver.cc:460] Webserver started at http://127.4.177.67:45123/ using document root <none> and password file <none>
I20260808 21:08:33.476553 20144 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:33.476629 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:33.476804 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:33.477919 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "524e7f3f517546b18fed2e628283fa39"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "7b848847d5deaaca3796b76ebbd75cc8"
server_key_iv: "e9ebeb7e3ba249af4b7310065ddcdde8"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.478327 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "524e7f3f517546b18fed2e628283fa39"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "7b848847d5deaaca3796b76ebbd75cc8"
server_key_iv: "e9ebeb7e3ba249af4b7310065ddcdde8"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.479854 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:08:33.480619 20160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.480800 20144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:33.480906 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "524e7f3f517546b18fed2e628283fa39"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "7b848847d5deaaca3796b76ebbd75cc8"
server_key_iv: "e9ebeb7e3ba249af4b7310065ddcdde8"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.480995 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:33.502291 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:33.504875 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:33.505062 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:33.505327 20144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:33.505659 20144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:33.505721 20144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.505789 20144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:33.505823 20144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.512104 20144 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40513
I20260808 21:08:33.512143 20281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40513 every 8 connection(s)
I20260808 21:08:33.512491 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:08:33.516216 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20144
I20260808 21:08:33.516319 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:08:33.516503 4805 external_mini_cluster.cc:1428] Setting key 51aea26dfff480e01dbc9d4491fd76e2
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:33.522921 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.513309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:51253 (local address 127.4.177.126:37635)
0808 21:08:33.513386 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.513389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.513585 (+ 196us) server_negotiation.cc:407] Connection header received
0808 21:08:33.514172 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.514174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.514188 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.514246 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:33.515336 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.515900 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.517159 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.517385 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.517477 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.519774 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:33.519775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:33.519776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:33.519784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:33.521371 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.521849 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.521851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.521851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.521915 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.522216 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.522217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.522217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.522345 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:33.522422 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.522733 (+ 311us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.522818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:08:33.523129 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.513237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:51253)
0808 21:08:33.513437 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.513487 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.514048 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.514320 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.514325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.514958 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:33.515226 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.515233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.516005 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.516006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.517049 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.517051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.517489 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.517490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.517638 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.518093 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:33.518102 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:33.519657 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:33.521466 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.521473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.521476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.521763 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.522005 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:33.522005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:33.522006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:33.522133 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:33.522425 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:33.522433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:33.522584 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.522697 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.522869 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260808 21:08:33.523607 20282 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:33.523705 20282 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:33.523849 20282 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:33.524205 19790 ts_manager.cc:194] Registered new tserver with Master: 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513)
I20260808 21:08:33.524539 19790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:51253
I20260808 21:08:33.528210 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:08:33.535853 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.529568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.4.177.126:37635)
0808 21:08:33.529633 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.529634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.529756 (+ 122us) server_negotiation.cc:407] Connection header received
0808 21:08:33.529816 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.529817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.529825 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.529866 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:33.530475 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.530929 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.531762 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.531936 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.531981 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.533251 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:33.533252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:33.533252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:33.533258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:33.534636 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.534953 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.534954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.534954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.535005 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:33.535280 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:33.535281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:33.535281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:33.535413 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:33.535488 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.535649 (+ 161us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.535738 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 21:08:33.537480 19790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:08:33.539148 19790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:08:33.545408 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40513 (local address 127.0.0.1:47804)
0808 21:08:33.541667 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.541711 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.541782 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.542030 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.542033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.542035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:33.542101 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.542102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543181 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.543182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.544195 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.544197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.545097 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.545098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.545185 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.545200 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.545277 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.545329 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":45,"threads_started":1}
I20260808 21:08:33.545421 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.4.177.67:40513)
0808 21:08:33.541855 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.541860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.541873 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:08:33.541912 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.541915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.541931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.542030 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:33.542191 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543064 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.544293 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.545013 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.545081 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.545135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.545304 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.545344 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260808 21:08:33.546173 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.0.0.1:54836)
0808 21:08:33.542125 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.542135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.542177 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.542439 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.542440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.542442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:33.542475 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.542476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543863 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.543864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.544828 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.544831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.545919 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.545920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.545985 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.545990 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.546046 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.546085 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":85,"threads_started":1}
I20260808 21:08:33.546289 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.0.0.1:32792)
0808 21:08:33.541845 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.541856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.541913 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.542239 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.542241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.542243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:33.542299 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.542302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543797 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.543799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.544849 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.544852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.546062 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.546063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.546114 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.546117 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.546162 (+ 45us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.546197 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":85,"threads_started":1}
I20260808 21:08:33.546299 20209 tablet_service.cc:1467] Processing CreateTablet for tablet 27fab338fad24f90ba930ba27ae3c836 (DEFAULT_TABLE table=test-table [id=fc53ac5ce60941d1b0272636411b5bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:33.546197 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.4.177.66:38427)
0808 21:08:33.542204 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.542208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.542219 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:08:33.542258 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.542259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.542276 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.542361 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:33.542561 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543762 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.544932 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.545822 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.545904 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.545960 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.546079 (+ 119us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.546117 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":120,"threads_started":1}
I20260808 21:08:33.546581 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.541618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.4.177.65:35989)
0808 21:08:33.541964 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.541969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.541980 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:08:33.542037 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.542040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.542059 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.542145 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:33.542387 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.543668 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.544950 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.545907 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.545974 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.546029 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.546454 (+ 425us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.546507 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
I20260808 21:08:33.546713 20209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fab338fad24f90ba930ba27ae3c836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:33.547242 20067 tablet_service.cc:1467] Processing CreateTablet for tablet 27fab338fad24f90ba930ba27ae3c836 (DEFAULT_TABLE table=test-table [id=fc53ac5ce60941d1b0272636411b5bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:33.547571 20067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fab338fad24f90ba930ba27ae3c836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:33.547559 19924 tablet_service.cc:1467] Processing CreateTablet for tablet 27fab338fad24f90ba930ba27ae3c836 (DEFAULT_TABLE table=test-table [id=fc53ac5ce60941d1b0272636411b5bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:33.547859 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fab338fad24f90ba930ba27ae3c836. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:33.549523 20299 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Bootstrap starting.
I20260808 21:08:33.550271 20299 tablet_bootstrap.cc:654] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:33.550757 20299 log.cc:824] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:33.550823 20300 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Bootstrap starting.
I20260808 21:08:33.551179 20301 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Bootstrap starting.
I20260808 21:08:33.551563 20300 tablet_bootstrap.cc:654] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:33.551884 20301 tablet_bootstrap.cc:654] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:33.552173 20300 log.cc:824] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:33.552518 20301 log.cc:824] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:33.553068 20300 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: No bootstrap required, opened a new log
I20260808 21:08:33.553184 20300 ts_tablet_manager.cc:1397] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:08:33.553373 20299 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: No bootstrap required, opened a new log
I20260808 21:08:33.553398 20301 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: No bootstrap required, opened a new log
I20260808 21:08:33.553470 20301 ts_tablet_manager.cc:1397] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:33.553478 20299 ts_tablet_manager.cc:1397] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260808 21:08:33.554905 20299 raft_consensus.cc:348] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.555063 20299 raft_consensus.cc:374] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:33.555107 20299 raft_consensus.cc:729] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524e7f3f517546b18fed2e628283fa39, State: Initialized, Role: FOLLOWER
I20260808 21:08:33.555251 20299 consensus_queue.cc:260] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.555372 20300 raft_consensus.cc:348] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.555516 20299 ts_tablet_manager.cc:1428] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:33.555562 20300 raft_consensus.cc:374] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:33.555617 20300 raft_consensus.cc:729] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386b2a0f586f4a819f071d48b2cdd234, State: Initialized, Role: FOLLOWER
I20260808 21:08:33.555680 20282 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
I20260808 21:08:33.555568 20301 raft_consensus.cc:348] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.555724 20301 raft_consensus.cc:374] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:33.555773 20301 raft_consensus.cc:729] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a523e37cfd147828101c003af349c1c, State: Initialized, Role: FOLLOWER
I20260808 21:08:33.555737 20300 consensus_queue.cc:260] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.555891 20301 consensus_queue.cc:260] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.556043 20300 ts_tablet_manager.cc:1428] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 21:08:33.556082 20139 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
I20260808 21:08:33.556193 20301 ts_tablet_manager.cc:1428] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:08:33.556321 19993 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
W20260808 21:08:33.595692 20140 tablet.cc:2369] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:08:33.635195 20307 raft_consensus.cc:482] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:33.635326 20307 raft_consensus.cc:504] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.635793 20307 leader_election.cc:288] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989), 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427)
I20260808 21:08:33.638665 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.635971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.4.177.67:52341)
0808 21:08:33.636053 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.636066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.636114 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.636321 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.636323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.636325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:33.636415 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.636417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.637072 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.637074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.637823 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.637825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.638394 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.638395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.638480 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.638483 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.638551 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.638595 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 21:08:33.638665 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.636093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:52341 (local address 127.4.177.65:35989)
0808 21:08:33.636166 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.636168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.636180 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:08:33.636213 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.636214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.636224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.636312 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:33.636491 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.636980 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.637921 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.638308 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.638397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.638405 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.638557 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.638595 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:08:33.638943 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.636273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:54327 (local address 127.4.177.66:38427)
0808 21:08:33.636328 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.636330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.636456 (+ 126us) server_negotiation.cc:407] Connection header received
0808 21:08:33.636516 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.636517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.636534 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.636579 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:33.636750 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.637217 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.638144 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.638536 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.638622 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.638630 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.638826 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.638875 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:08:33.639015 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fab338fad24f90ba930ba27ae3c836" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a523e37cfd147828101c003af349c1c" is_pre_election: true
I20260808 21:08:33.639212 19947 raft_consensus.cc:2454] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 0.
I20260808 21:08:33.639227 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.636141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.4.177.67:54327)
0808 21:08:33.636374 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.636391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:33.636461 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:33.636590 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:33.636591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:33.636592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:33.636656 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.636657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.637306 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.637307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.638049 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:33.638050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.638624 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.638625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:33.638708 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.638713 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:08:33.639066 (+ 353us) client_negotiation.cc:240] Negotiation successful
0808 21:08:33.639122 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":146,"threads_started":1}
I20260808 21:08:33.639468 20162 leader_election.cc:302] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524e7f3f517546b18fed2e628283fa39, 8a523e37cfd147828101c003af349c1c; no voters:
I20260808 21:08:33.639618 20307 raft_consensus.cc:2781] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:08:33.639590 20088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fab338fad24f90ba930ba27ae3c836" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234" is_pre_election: true
I20260808 21:08:33.639668 20307 raft_consensus.cc:482] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:33.639698 20307 raft_consensus.cc:3037] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:33.639729 20088 raft_consensus.cc:2454] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 0.
I20260808 21:08:33.640558 20307 raft_consensus.cc:504] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.640689 20307 leader_election.cc:288] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 election: Requested vote from peers 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989), 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427)
I20260808 21:08:33.640887 20088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fab338fad24f90ba930ba27ae3c836" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234"
I20260808 21:08:33.640905 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fab338fad24f90ba930ba27ae3c836" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a523e37cfd147828101c003af349c1c"
I20260808 21:08:33.640971 19947 raft_consensus.cc:3037] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:33.640969 20088 raft_consensus.cc:3037] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:33.641876 19947 raft_consensus.cc:2454] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 1.
I20260808 21:08:33.641876 20088 raft_consensus.cc:2454] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 1.
I20260808 21:08:33.642097 20162 leader_election.cc:302] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 524e7f3f517546b18fed2e628283fa39, 8a523e37cfd147828101c003af349c1c; no voters:
I20260808 21:08:33.642189 20307 raft_consensus.cc:2781] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:33.642350 20307 raft_consensus.cc:686] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 1 LEADER]: Becoming Leader. State: Replica: 524e7f3f517546b18fed2e628283fa39, State: Running, Role: LEADER
I20260808 21:08:33.642421 20307 consensus_queue.cc:237] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:33.643361 19788 catalog_manager.cc:5495] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524e7f3f517546b18fed2e628283fa39 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "524e7f3f517546b18fed2e628283fa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } health_report { overall_health: UNKNOWN } } }
I20260808 21:08:33.660900 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.658332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.4.177.67:40513)
0808 21:08:33.658393 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:33.658395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:33.658404 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:33.658489 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:33.658490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:33.658508 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:33.658610 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:08:33.658736 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.659315 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.660083 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:33.660298 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:33.660375 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:33.660533 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:08:33.660644 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:08:33.660730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:33.660776 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 21:08:33.660796 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:08:33.662783 20088 raft_consensus.cc:1261] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Refusing update from remote peer 524e7f3f517546b18fed2e628283fa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:33.662823 19947 raft_consensus.cc:1261] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Refusing update from remote peer 524e7f3f517546b18fed2e628283fa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:33.663269 20307 consensus_queue.cc:1035] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:33.663424 20311 consensus_queue.cc:1035] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:33.665612 20314 mvcc.cc:205] Tried to move back new op lower bound from 7316370692760137728 to 7316370692680056832. Current Snapshot: MvccSnapshot[applied={T|T < 7316370692760137728}]
I20260808 21:08:33.666591 20315 mvcc.cc:205] Tried to move back new op lower bound from 7316370692760137728 to 7316370692680056832. Current Snapshot: MvccSnapshot[applied={T|T < 7316370692760137728}]
I20260808 21:08:33.666697 20316 mvcc.cc:205] Tried to move back new op lower bound from 7316370692760137728 to 7316370692680056832. Current Snapshot: MvccSnapshot[applied={T|T < 7316370692760137728 or (T in {7316370692760137728})}]
I20260808 21:08:33.668062 19788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:33.668155 19788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:33.668838 19788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Sending DeleteTablet for 3 replicas of tablet 27fab338fad24f90ba930ba27ae3c836
I20260808 21:08:33.669135 19924 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fab338fad24f90ba930ba27ae3c836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:33 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:32792
I20260808 21:08:33.669170 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19855
I20260808 21:08:33.669188 20067 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fab338fad24f90ba930ba27ae3c836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:33 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54836
I20260808 21:08:33.670841 20324 tablet_replica.cc:330] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: stopping tablet replica
I20260808 21:08:33.670974 20324 raft_consensus.cc:2229] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:33.671082 20324 raft_consensus.cc:2258] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:33.671512 20324 ts_tablet_manager.cc:1905] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:33.673372 20209 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fab338fad24f90ba930ba27ae3c836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:33 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47804
I20260808 21:08:33.673589 20325 tablet_replica.cc:330] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: stopping tablet replica
I20260808 21:08:33.673727 20325 raft_consensus.cc:2229] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:08:33.673777 20324 ts_tablet_manager.cc:1918] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:33.673859 20324 log.cc:1196] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/27fab338fad24f90ba930ba27ae3c836
I20260808 21:08:33.673971 20325 raft_consensus.cc:2258] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:33.674149 20324 ts_tablet_manager.cc:1939] T 27fab338fad24f90ba930ba27ae3c836 P 386b2a0f586f4a819f071d48b2cdd234: Deleting consensus metadata
I20260808 21:08:33.674561 20325 ts_tablet_manager.cc:1905] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:33.674556 19773 catalog_manager.cc:4841] TS 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427): tablet 27fab338fad24f90ba930ba27ae3c836 (table test-table [id=fc53ac5ce60941d1b0272636411b5bad]) successfully deleted
I20260808 21:08:33.676484 20325 ts_tablet_manager.cc:1918] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:33.676635 20325 log.cc:1196] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/27fab338fad24f90ba930ba27ae3c836
I20260808 21:08:33.677004 20325 ts_tablet_manager.cc:1939] T 27fab338fad24f90ba930ba27ae3c836 P 524e7f3f517546b18fed2e628283fa39: Deleting consensus metadata
I20260808 21:08:33.677423 19775 catalog_manager.cc:4841] TS 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513): tablet 27fab338fad24f90ba930ba27ae3c836 (table test-table [id=fc53ac5ce60941d1b0272636411b5bad]) successfully deleted
W20260808 21:08:33.677608 19773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:35989 (error 108)
I20260808 21:08:33.677914 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19999
I20260808 21:08:33.678092 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:33.677969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.0.0.1:32798)
0808 21:08:33.678017 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:33.678053 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:08:33.678968 19773 catalog_manager.cc:4584] TS 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27fab338fad24f90ba930ba27ae3c836: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35989: connect: Connection refused (error 111)
I20260808 21:08:33.688344 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20144
I20260808 21:08:33.693785 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 19755
I20260808 21:08:33.700764 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--webserver_interface=127.4.177.126
--webserver_port=46549
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:33.787154 20327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:33.787387 20327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:33.787426 20327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:33.788576 20327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:33.788640 20327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:33.788662 20327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:33.788683 20327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:33.788710 20327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:33.790174 20327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:33.790437 20327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:33.790674 20327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:33.801050 20337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.801030 20334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.801055 20335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.801229 20327 server_base.cc:1034] running on GCE node
I20260808 21:08:33.801656 20327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:33.801983 20327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:33.803194 20327 hybrid_clock.cc:648] HybridClock initialized: now 1786223313803180 us; error 45 us; skew 500 ppm
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:33.808761 20327 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:33.809247 20327 webserver.cc:460] Webserver started at http://127.4.177.126:46549/ using document root <none> and password file <none>
I20260808 21:08:33.809512 20327 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:33.809607 20327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:33.810971 20327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:33.811697 20344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.811877 20327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.811970 20327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b71e8a690517484ba3e5aae7e916020e"
format_stamp: "Formatted at 2026-08-08 21:08:32 on dist-test-slave-chsk"
server_key: "0f9474c282405f51b2c9818b26805887"
server_key_iv: "7f35880aae7e37e2c16ad6cfb7a998b4"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.812389 20327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:33.834152 20327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:33.836256 20327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:33.836421 20327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:33.840827 20327 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:37635
I20260808 21:08:33.840883 20399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:37635 every 8 connection(s)
I20260808 21:08:33.841295 20327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:33.842976 20400 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:08:33.843684 20400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap starting.
I20260808 21:08:33.846443 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20327
I20260808 21:08:33.846763 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:35989
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=34145
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:33.852550 20400 log.cc:824] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:33.854441 20400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:33.854758 20400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap complete.
I20260808 21:08:33.857170 20400 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.857431 20400 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Initialized, Role: FOLLOWER
I20260808 21:08:33.857642 20400 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.857739 20400 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:33.857798 20400 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:33.857847 20400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:08:33.858840 20400 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.858973 20400 leader_election.cc:302] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71e8a690517484ba3e5aae7e916020e; no voters:
I20260808 21:08:33.859198 20400 leader_election.cc:288] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 21:08:33.859234 20404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:08:33.859354 20404 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 LEADER]: Becoming Leader. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Running, Role: LEADER
I20260808 21:08:33.859491 20400 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:33.859519 20404 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:33.859933 20406 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71e8a690517484ba3e5aae7e916020e. Latest consensus state: current_term: 2 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:33.860025 20406 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:33.860248 20405 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:33.860338 20405 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:33.861390 20421 catalog_manager.cc:1259] Loaded cluster ID: f28a695b80444e2baf5d189d6a460229
I20260808 21:08:33.861457 20421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: loading cluster ID for follower catalog manager: success
W20260808 21:08:33.861685 20421 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260808 21:08:33.861752 20421 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260808 21:08:33.861806 20412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:33.862159 20412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:33.862265 20412 catalog_manager.cc:1259] Loaded cluster ID: f28a695b80444e2baf5d189d6a460229
I20260808 21:08:33.862323 20412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:08:33.862511 20412 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 21:08:33.865247 20412 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 21:08:33.867125 20412 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:33.867374 20412 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Loaded TSK: 0
I20260808 21:08:33.867594 20412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 21:08:33.942651 20402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:33.942884 20402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:33.942924 20402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:33.944092 20402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:33.944159 20402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:33.944203 20402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:33.945684 20402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=34145
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:33.945960 20402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:33.946199 20402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:33.952199 20429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.952299 20428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:33.952219 20431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:33.952468 20402 server_base.cc:1034] running on GCE node
I20260808 21:08:33.952718 20402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:33.952996 20402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:33.954151 20402 hybrid_clock.cc:648] HybridClock initialized: now 1786223313954135 us; error 33 us; skew 500 ppm
Aug 08 21:08:33 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:33.956765 20402 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:33.957267 20402 webserver.cc:460] Webserver started at http://127.4.177.65:34145/ using document root <none> and password file <none>
I20260808 21:08:33.957541 20402 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:33.957600 20402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:33.959007 20402 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:08:33.959873 20439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:33.960040 20402 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:33.960132 20402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "8a523e37cfd147828101c003af349c1c"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "cf9d108de3cf812105c60707077d2da9"
server_key_iv: "64bfb896ab54c7d30202ee6a2136de24"
server_key_version: "encryptionkey@0"
I20260808 21:08:33.960572 20402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.017132 20402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.019632 20402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.019814 20402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.020076 20402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:34.020588 20446 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 21:08:34.022179 20402 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 21:08:34.022276 20402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 21:08:34.022348 20402 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 21:08:34.023088 20402 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 21:08:34.023144 20402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 21:08:34.023252 20446 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Bootstrap starting.
I20260808 21:08:34.030576 20402 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35989
I20260808 21:08:34.030676 20559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35989 every 8 connection(s)
I20260808 21:08:34.031039 20402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:08:34.032543 20446 log.cc:824] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.033182 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20402
I20260808 21:08:34.033552 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:38427
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=40765
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:34.037382 20446 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:34.037758 20446 tablet_bootstrap.cc:492] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Bootstrap complete.
I20260808 21:08:34.038873 20446 ts_tablet_manager.cc:1397] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:34.039927 20446 raft_consensus.cc:348] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:34.040182 20446 raft_consensus.cc:729] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a523e37cfd147828101c003af349c1c, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.040343 20446 consensus_queue.cc:260] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } }
I20260808 21:08:34.040647 20446 ts_tablet_manager.cc:1428] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:08:34.044332 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.031781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:38103)
0808 21:08:34.032050 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.032095 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.032567 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.033133 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.033139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.033595 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:34.033888 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.033903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.035043 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.035044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.036081 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.036084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.036797 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.036799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.036976 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.037634 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:34.037645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:34.039769 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:34.042625 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.042632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.042633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.042862 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.043230 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.043231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.043231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.043330 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.043876 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:34.043880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:34.043987 (+ 107us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.044064 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.044186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260808 21:08:34.044608 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.031965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:38103 (local address 127.4.177.126:37635)
0808 21:08:34.032192 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.032202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.032250 (+ 48us) server_negotiation.cc:407] Connection header received
0808 21:08:34.032801 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.032808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.032870 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.033016 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:34.034037 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.034906 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.036196 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.036650 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.036800 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.039916 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:34.039920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:34.039921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:34.039936 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:34.042532 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.042954 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.042959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.042959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.043115 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.043408 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.043409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.043410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.043786 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:34.043890 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.044168 (+ 278us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.044351 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260808 21:08:34.045275 20560 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:34.045394 20560 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:34.045617 20560 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:34.046409 20362 ts_manager.cc:194] Registered new tserver with Master: 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989)
I20260808 21:08:34.047501 20362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:38103
I20260808 21:08:34.048548 20560 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
I20260808 21:08:34.054378 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.046993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.0.0.1:32804)
0808 21:08:34.050844 (+ 3851us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.050880 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.051001 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.051398 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.051401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.051404 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:34.051489 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.051491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.052559 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.052560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.053370 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.053373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.054035 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.054036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.054127 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.054140 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.054247 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.054307 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"thread_start_us":83,"threads_started":1}
I20260808 21:08:34.054376 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.047093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.4.177.65:35989)
0808 21:08:34.047271 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.047275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.050961 (+ 3686us) server_negotiation.cc:407] Connection header received
0808 21:08:34.051195 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.051197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.051215 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.051321 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:34.051550 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.052471 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.053467 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.053931 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.054030 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.054088 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.054263 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.054304 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260808 21:08:34.054931 20488 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fab338fad24f90ba930ba27ae3c836 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:33 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:32804
I20260808 21:08:34.055503 20569 tablet_replica.cc:330] stopping tablet replica
I20260808 21:08:34.055629 20569 raft_consensus.cc:2229] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:34.055720 20569 raft_consensus.cc:2258] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:34.056032 20569 ts_tablet_manager.cc:1905] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:34.058215 20569 ts_tablet_manager.cc:1918] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:34.058303 20569 log.cc:1196] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/27fab338fad24f90ba930ba27ae3c836
I20260808 21:08:34.058524 20569 ts_tablet_manager.cc:1939] T 27fab338fad24f90ba930ba27ae3c836 P 8a523e37cfd147828101c003af349c1c: Deleting consensus metadata
I20260808 21:08:34.059060 20346 catalog_manager.cc:4841] TS 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989): tablet 27fab338fad24f90ba930ba27ae3c836 (table test-table [id=fc53ac5ce60941d1b0272636411b5bad]) successfully deleted
W20260808 21:08:34.127885 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.128080 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.128104 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.129238 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.129279 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.129305 20565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:34.130739 20565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=40765
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.130924 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.131094 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.135864 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.135854 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.135856 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.136036 20565 server_base.cc:1034] running on GCE node
I20260808 21:08:34.136293 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.136526 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.137694 20565 hybrid_clock.cc:648] HybridClock initialized: now 1786223314137679 us; error 27 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.140096 20565 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:34.140535 20565 webserver.cc:460] Webserver started at http://127.4.177.66:40765/ using document root <none> and password file <none>
I20260808 21:08:34.140764 20565 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.140846 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.142231 20565 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.142820 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.143004 20565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:34.143113 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "386b2a0f586f4a819f071d48b2cdd234"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "18ab915c0767e65ac0d53a5d8589a835"
server_key_iv: "efdbecadcde0997f3c5a711f6c29a7ed"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.143484 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.159898 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.162047 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.162210 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.162421 20565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:34.162710 20565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:34.162765 20565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.162806 20565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:34.162833 20565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.168776 20565 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38427
I20260808 21:08:34.168820 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38427 every 8 connection(s)
I20260808 21:08:34.169144 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:08:34.170536 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20565
I20260808 21:08:34.170827 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:40513
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=45123
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:34.179345 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.169830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:42319)
0808 21:08:34.169999 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.170033 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.170648 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.170903 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.170907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.171778 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:34.172038 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.172044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.172735 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.172736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.173511 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.173513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.173952 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.173953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.174079 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.174593 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:34.174601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:34.176069 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:34.177989 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.177995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.177997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.178212 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.178431 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.178432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.178432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.178525 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.178832 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:34.178836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:34.178941 (+ 105us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.179019 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.179133 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260808 21:08:34.179337 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.169841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:42319 (local address 127.4.177.126:37635)
0808 21:08:34.169890 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.169892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.170112 (+ 220us) server_negotiation.cc:407] Connection header received
0808 21:08:34.170765 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.170767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.170778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.170829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:34.172135 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.172624 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.173626 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.173854 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.173941 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.176163 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:34.176165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:34.176165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:34.176174 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:34.177894 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.178287 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.178289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.178289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.178358 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.178601 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.178602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.178602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.178759 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:34.178842 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.179070 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.179193 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:08:34.179847 20703 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:34.179955 20703 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:34.180158 20703 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:34.180580 20362 ts_manager.cc:194] Registered new tserver with Master: 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427)
I20260808 21:08:34.181120 20362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:42319
W20260808 21:08:34.254174 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.254364 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.254387 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.255539 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.255578 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.255604 20707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:34.257064 20707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:40513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=45123
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.257237 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.257414 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.262356 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.262334 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.262423 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.262449 20707 server_base.cc:1034] running on GCE node
I20260808 21:08:34.262763 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.262997 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.264144 20707 hybrid_clock.cc:648] HybridClock initialized: now 1786223314264123 us; error 37 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.266690 20707 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:34.267062 20707 webserver.cc:460] Webserver started at http://127.4.177.67:45123/ using document root <none> and password file <none>
I20260808 21:08:34.267295 20707 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.267355 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.268581 20707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.269110 20723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.269276 20707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:34.269381 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "524e7f3f517546b18fed2e628283fa39"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "7b848847d5deaaca3796b76ebbd75cc8"
server_key_iv: "e9ebeb7e3ba249af4b7310065ddcdde8"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.269711 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.279707 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.281873 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.282045 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.282258 20707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:34.282541 20707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:34.282593 20707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.282632 20707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:34.282661 20707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.288429 20707 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40513
I20260808 21:08:34.288477 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40513 every 8 connection(s)
I20260808 21:08:34.288790 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:34.296697 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20707
I20260808 21:08:34.297456 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.289461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33573 (local address 127.4.177.126:37635)
0808 21:08:34.289525 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.289527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.289706 (+ 179us) server_negotiation.cc:407] Connection header received
0808 21:08:34.290169 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.290170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.290179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.290241 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:34.291085 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.291589 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.292490 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.292668 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.292717 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.294728 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:34.294729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:34.294729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:34.294736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:34.296019 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.296385 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.296386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.296386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.296435 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.296675 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.296676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.296677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.296826 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:34.296899 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.297215 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.297321 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:08:34.297603 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.289427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:33573)
0808 21:08:34.289581 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.289619 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.290084 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.290285 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.290287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.290823 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:34.290997 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.291000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.291663 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.291663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.292415 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.292417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.292737 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.292738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.292829 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.293322 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:34.293330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:34.294646 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:34.296096 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.296101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.296103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.296319 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.296510 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:34.296511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:34.296511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:34.296609 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:34.296931 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:34.296938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:34.297053 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.297162 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.297343 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260808 21:08:34.298053 20847 heartbeater.cc:347] Connected to a master server at 127.4.177.126:37635
I20260808 21:08:34.298161 20847 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:34.298367 20847 heartbeater.cc:511] Master 127.4.177.126:37635 requested a full tablet report, sending...
I20260808 21:08:34.298789 20362 ts_manager.cc:194] Registered new tserver with Master: 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513)
I20260808 21:08:34.299291 20362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:33573
I20260808 21:08:34.303803 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.297167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.4.177.126:37635)
0808 21:08:34.297341 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.297343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.297421 (+ 78us) server_negotiation.cc:407] Connection header received
0808 21:08:34.297470 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.297471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.297485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.297556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:34.298438 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.299186 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.300229 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.300413 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.300467 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.301496 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:34.301497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:34.301497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:34.301503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:34.302764 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.303073 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.303073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.303074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.303125 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:34.303381 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:34.303383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:34.303383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:34.303497 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:34.303618 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.303671 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.303739 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":54,"threads_started":1}
I20260808 21:08:34.304491 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:08:34.306816 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.304807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.4.177.126:37635)
0808 21:08:34.304840 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.304841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.304939 (+ 98us) server_negotiation.cc:407] Connection header received
0808 21:08:34.304996 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.304997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.305023 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.305083 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:08:34.305165 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.305586 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.306253 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.306427 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.306476 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.306595 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:08:34.306673 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:08:34.306724 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.306757 (+ 33us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.306772 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:08:34.307077 20362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:08:34.308671 20362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:08:34.311020 20488 tablet_service.cc:1467] Processing CreateTablet for tablet b670cddbb9f64485977e32bf411235d6 (DEFAULT_TABLE table=test-table [id=d747f22893ab4b32a17d38f21488a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:34.311270 20488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b670cddbb9f64485977e32bf411235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:34.312937 20446 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Bootstrap starting.
I20260808 21:08:34.313273 20446 tablet_bootstrap.cc:654] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:34.314008 20446 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: No bootstrap required, opened a new log
I20260808 21:08:34.314064 20446 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.314183 20446 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.314235 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.310545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.0.0.1:54850)
0808 21:08:34.310667 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.310677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.310711 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.311125 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.311126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.311127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:34.311190 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.311191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.312296 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.312297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.313091 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.313094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.313984 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.313985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.314061 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.314065 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.314129 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.314173 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 21:08:34.314251 20446 raft_consensus.cc:374] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:34.314298 20446 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a523e37cfd147828101c003af349c1c, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.314354 20446 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.314450 20446 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.314460 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.310772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.4.177.66:38427)
0808 21:08:34.310941 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.310947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.310957 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:08:34.311001 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.311003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.311021 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.311113 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:34.311256 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.312179 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.313169 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.313901 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.313969 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.314024 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.314334 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.314383 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260808 21:08:34.314888 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.310545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40513 (local address 127.0.0.1:47818)
0808 21:08:34.310775 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.310786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.310841 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.311221 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.311222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.311224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:34.311269 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.311270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.312592 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.312593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.313707 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.313711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.314639 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.314640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.314704 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.314707 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.314766 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.314811 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":22,"thread_start_us":126,"threads_started":1}
I20260808 21:08:34.315130 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.310650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.4.177.67:40513)
0808 21:08:34.310989 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.310993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.311002 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:34.311043 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.311046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.311061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.311134 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:34.311339 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.312458 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.313822 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.314478 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.314546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.314602 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.315019 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.315062 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":88,"threads_started":1}
I20260808 21:08:34.315435 20634 tablet_service.cc:1467] Processing CreateTablet for tablet b670cddbb9f64485977e32bf411235d6 (DEFAULT_TABLE table=test-table [id=d747f22893ab4b32a17d38f21488a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:34.315692 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b670cddbb9f64485977e32bf411235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:34.318284 20857 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Bootstrap starting.
I20260808 21:08:34.319047 20857 tablet_bootstrap.cc:654] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:34.318922 20775 tablet_service.cc:1467] Processing CreateTablet for tablet b670cddbb9f64485977e32bf411235d6 (DEFAULT_TABLE table=test-table [id=d747f22893ab4b32a17d38f21488a6f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:34.319301 20775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b670cddbb9f64485977e32bf411235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:34.319620 20857 log.cc:824] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.320339 20857 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: No bootstrap required, opened a new log
I20260808 21:08:34.320425 20857 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:34.321640 20859 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Bootstrap starting.
I20260808 21:08:34.321915 20857 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.322132 20857 raft_consensus.cc:374] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:34.322175 20857 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386b2a0f586f4a819f071d48b2cdd234, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.322371 20859 tablet_bootstrap.cc:654] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:34.322360 20857 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.322693 20857 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:34.322706 20703 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
I20260808 21:08:34.322990 20859 log.cc:824] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.323833 20859 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: No bootstrap required, opened a new log
I20260808 21:08:34.323886 20859 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:34.325139 20859 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.325268 20859 raft_consensus.cc:374] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:34.325289 20859 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524e7f3f517546b18fed2e628283fa39, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.325400 20859 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.325659 20859 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:34.325685 20847 heartbeater.cc:503] Master 127.4.177.126:37635 was elected leader, sending a full tablet report...
I20260808 21:08:34.364720 20862 raft_consensus.cc:482] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:34.364847 20862 raft_consensus.cc:504] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.365249 20862 leader_election.cc:288] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427), 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989)
I20260808 21:08:34.368197 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.365458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.4.177.67:37419)
0808 21:08:34.365518 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.365531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.365604 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.365779 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.365780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.365782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:34.365858 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.365860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.366489 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.366490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.367336 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.367338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.367915 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.367916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.367998 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.368002 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.368062 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.368103 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:08:34.368194 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.365590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:37419 (local address 127.4.177.66:38427)
0808 21:08:34.365661 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.365663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.365673 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:08:34.365702 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.365704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.365713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.365774 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:34.365941 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.366407 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.367423 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.367835 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.367911 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.367921 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.368066 (+ 145us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.368108 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260808 21:08:34.368520 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.365690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:43243 (local address 127.4.177.65:35989)
0808 21:08:34.365750 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.365752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.365845 (+ 93us) server_negotiation.cc:407] Connection header received
0808 21:08:34.365910 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.365912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.365925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.365980 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:34.366131 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.366591 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.367589 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.368131 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.368191 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.368198 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.368416 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.368454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:08:34.368558 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234" is_pre_election: true
I20260808 21:08:34.368718 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.365577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.4.177.67:43243)
0808 21:08:34.365769 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.365778 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.365855 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.365984 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:34.365984 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:34.365986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:34.366058 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.366059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.366670 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.366671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.367501 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:34.367504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.368231 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.368232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:34.368314 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.368319 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:08:34.368609 (+ 290us) client_negotiation.cc:240] Negotiation successful
0808 21:08:34.368652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":118,"threads_started":1}
I20260808 21:08:34.368705 20654 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 0.
I20260808 21:08:34.368944 20725 leader_election.cc:302] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386b2a0f586f4a819f071d48b2cdd234, 524e7f3f517546b18fed2e628283fa39; no voters:
I20260808 21:08:34.369096 20862 raft_consensus.cc:2781] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:08:34.369185 20862 raft_consensus.cc:482] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:34.369213 20862 raft_consensus.cc:3037] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:34.369513 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a523e37cfd147828101c003af349c1c" is_pre_election: true
I20260808 21:08:34.369655 20510 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 0.
I20260808 21:08:34.370133 20862 raft_consensus.cc:504] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.370272 20862 leader_election.cc:288] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 election: Requested vote from peers 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427), 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989)
I20260808 21:08:34.370460 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234"
I20260808 21:08:34.370512 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "524e7f3f517546b18fed2e628283fa39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a523e37cfd147828101c003af349c1c"
I20260808 21:08:34.370558 20654 raft_consensus.cc:3037] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:34.370625 20510 raft_consensus.cc:3037] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:34.371402 20654 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 1.
I20260808 21:08:34.371447 20510 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 524e7f3f517546b18fed2e628283fa39 in term 1.
I20260808 21:08:34.371590 20725 leader_election.cc:302] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 386b2a0f586f4a819f071d48b2cdd234, 524e7f3f517546b18fed2e628283fa39; no voters:
I20260808 21:08:34.371672 20862 raft_consensus.cc:2781] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:34.371820 20862 raft_consensus.cc:686] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 1 LEADER]: Becoming Leader. State: Replica: 524e7f3f517546b18fed2e628283fa39, State: Running, Role: LEADER
I20260808 21:08:34.371924 20862 consensus_queue.cc:237] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.372741 20362 catalog_manager.cc:5495] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 524e7f3f517546b18fed2e628283fa39 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "524e7f3f517546b18fed2e628283fa39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } health_report { overall_health: UNKNOWN } } }
I20260808 21:08:34.386802 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.384855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.4.177.67:40513)
0808 21:08:34.384901 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:34.384902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:34.384927 (+ 25us) server_negotiation.cc:407] Connection header received
0808 21:08:34.384989 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:34.384990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:34.385003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:34.385073 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:08:34.385158 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.385584 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.386225 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:34.386394 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:34.386454 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:34.386570 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:08:34.386649 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:08:34.386699 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:34.386730 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 21:08:34.386742 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:08:34.388597 20510 raft_consensus.cc:1261] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Refusing update from remote peer 524e7f3f517546b18fed2e628283fa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:34.388619 20654 raft_consensus.cc:1261] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Refusing update from remote peer 524e7f3f517546b18fed2e628283fa39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:34.388909 20862 consensus_queue.cc:1035] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:34.389050 20865 consensus_queue.cc:1035] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:34.391074 20867 mvcc.cc:205] Tried to move back new op lower bound from 7316370695733272576 to 7316370695668154368. Current Snapshot: MvccSnapshot[applied={T|T < 7316370695733272576}]
I20260808 21:08:34.391340 20868 mvcc.cc:205] Tried to move back new op lower bound from 7316370695733272576 to 7316370695668154368. Current Snapshot: MvccSnapshot[applied={T|T < 7316370695733272576}]
I20260808 21:08:34.392056 20870 mvcc.cc:205] Tried to move back new op lower bound from 7316370695733272576 to 7316370695668154368. Current Snapshot: MvccSnapshot[applied={T|T < 7316370695733272576}]
I20260808 21:08:34.393446 20361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:34.393543 20361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:34.394260 20361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Sending DeleteTablet for 3 replicas of tablet b670cddbb9f64485977e32bf411235d6
I20260808 21:08:34.394585 20775 tablet_service.cc:1514] Processing DeleteTablet for tablet b670cddbb9f64485977e32bf411235d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:34 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:47818
I20260808 21:08:34.394632 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet b670cddbb9f64485977e32bf411235d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:34 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54850
I20260808 21:08:34.394789 20488 tablet_service.cc:1514] Processing DeleteTablet for tablet b670cddbb9f64485977e32bf411235d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:34 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:32804
I20260808 21:08:34.394835 20877 tablet_replica.cc:330] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: stopping tablet replica
I20260808 21:08:34.394879 20569 tablet_replica.cc:330] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: stopping tablet replica
I20260808 21:08:34.394920 20569 raft_consensus.cc:2229] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:34.394920 20877 raft_consensus.cc:2229] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:08:34.394969 20569 raft_consensus.cc:2258] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:34.395002 20877 raft_consensus.cc:2258] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:34.395190 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20402
I20260808 21:08:34.396905 20877 ts_tablet_manager.cc:1905] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:34.398932 20877 ts_tablet_manager.cc:1918] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:34.398998 20877 log.cc:1196] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/b670cddbb9f64485977e32bf411235d6
I20260808 21:08:34.399289 20877 ts_tablet_manager.cc:1939] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Deleting consensus metadata
I20260808 21:08:34.399660 20348 catalog_manager.cc:4841] TS 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513): tablet b670cddbb9f64485977e32bf411235d6 (table test-table [id=d747f22893ab4b32a17d38f21488a6f2]) successfully deleted
W20260808 21:08:34.405210 20346 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:35989 (error 108)
I20260808 21:08:34.405308 20878 tablet_replica.cc:330] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: stopping tablet replica
I20260808 21:08:34.405427 20878 raft_consensus.cc:2229] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:34.405550 20878 raft_consensus.cc:2258] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 21:08:34.405885 20853 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:08:34.405682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.0.0.1:32820)
0808 21:08:34.405734 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.405746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.405796 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.405838 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35989: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 21:08:34.406028 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20565
I20260808 21:08:34.406087 20878 ts_tablet_manager.cc:1905] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260808 21:08:34.406086 20346 catalog_manager.cc:4584] TS 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b670cddbb9f64485977e32bf411235d6: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35989: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260808 21:08:34.417176 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20707
I20260808 21:08:34.417208 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.417086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.0.0.1:54862)
0808 21:08:34.417136 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.417180 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.66:38427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 21:08:34.423411 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20327
I20260808 21:08:34.430411 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--webserver_interface=127.4.177.126
--webserver_port=46549
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:37635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:34.511762 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.511962 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.511982 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.513093 20880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:34.513134 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.513149 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.513160 20880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:34.513211 20880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:34.514601 20880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:37635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.514834 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.515069 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.519721 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.519760 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.519760 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.520082 20880 server_base.cc:1034] running on GCE node
I20260808 21:08:34.520336 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.520658 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.521845 20880 hybrid_clock.cc:648] HybridClock initialized: now 1786223314521820 us; error 34 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.524421 20880 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:34.524892 20880 webserver.cc:460] Webserver started at http://127.4.177.126:46549/ using document root <none> and password file <none>
I20260808 21:08:34.525136 20880 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.525184 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.526504 20880 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:08:34.527179 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.527383 20880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:08:34.527458 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b71e8a690517484ba3e5aae7e916020e"
format_stamp: "Formatted at 2026-08-08 21:08:32 on dist-test-slave-chsk"
server_key: "0f9474c282405f51b2c9818b26805887"
server_key_iv: "7f35880aae7e37e2c16ad6cfb7a998b4"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.527767 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.538985 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.540709 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.540814 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.545009 20880 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:37635
I20260808 21:08:34.545040 20952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:37635 every 8 connection(s)
I20260808 21:08:34.545464 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:34.547010 20953 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:08:34.547713 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap starting.
I20260808 21:08:34.555548 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20880
I20260808 21:08:34.555845 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:35989
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=34145
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:08:34.556591 20953 log.cc:824] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.558810 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:34.559116 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e: Bootstrap complete.
I20260808 21:08:34.561478 20953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:34.561709 20953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.561896 20953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:34.561976 20953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:34.562077 20953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:34.562134 20953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 2 FOLLOWER]: Advancing to term 3
I20260808 21:08:34.563090 20953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:34.563220 20953 leader_election.cc:302] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71e8a690517484ba3e5aae7e916020e; no voters:
I20260808 21:08:34.563426 20953 leader_election.cc:288] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 21:08:34.563464 20957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 21:08:34.563611 20957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [term 3 LEADER]: Becoming Leader. State: Replica: b71e8a690517484ba3e5aae7e916020e, State: Running, Role: LEADER
I20260808 21:08:34.563761 20957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } }
I20260808 21:08:34.563766 20953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:34.564608 20959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71e8a690517484ba3e5aae7e916020e. Latest consensus state: current_term: 3 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:34.564689 20959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:34.564880 20967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:34.565024 20958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b71e8a690517484ba3e5aae7e916020e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71e8a690517484ba3e5aae7e916020e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37635 } } }
I20260808 21:08:34.565114 20958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71e8a690517484ba3e5aae7e916020e [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:34.565778 20967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:34.565927 20967 catalog_manager.cc:1259] Loaded cluster ID: f28a695b80444e2baf5d189d6a460229
I20260808 21:08:34.566130 20967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 21:08:34.566432 20967 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7c08849ad google::LogMessage::Fail() at ??:0
@ 0x7ff7c088788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7c08844de google::LogMessage::Flush() at ??:0
@ 0x7ff7c0885a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff7c87a50f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff7c87ae8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff7c27520d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff7c2747954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff7c35ab6db start_thread at ??:0
@ 0x7ff7bf38261f clone at ??:0
W20260808 21:08:34.653575 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.653822 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.653863 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.655025 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.655097 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.655143 20956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:34.656663 20956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:35989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=34145
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.656937 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.657181 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.662905 20993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.662923 20992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.663204 20956 server_base.cc:1034] running on GCE node
W20260808 21:08:34.663347 20995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.663581 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.663848 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.665007 20956 hybrid_clock.cc:648] HybridClock initialized: now 1786223314664994 us; error 31 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.667673 20956 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:34.668074 20956 webserver.cc:460] Webserver started at http://127.4.177.65:34145/ using document root <none> and password file <none>
I20260808 21:08:34.668334 20956 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.668383 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.670310 20956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260808 21:08:34.671253 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.671540 20956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.671654 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "8a523e37cfd147828101c003af349c1c"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "cf9d108de3cf812105c60707077d2da9"
server_key_iv: "64bfb896ab54c7d30202ee6a2136de24"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.672106 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.688284 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.690963 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.691118 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.691423 20956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:34.691905 21010 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 21:08:34.693414 20956 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 21:08:34.693478 20956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 21:08:34.693513 20956 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 21:08:34.694398 20956 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 21:08:34.694450 20956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 21:08:34.694536 21010 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Bootstrap starting.
I20260808 21:08:34.701658 20956 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35989
I20260808 21:08:34.701741 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35989 every 8 connection(s)
I20260808 21:08:34.702114 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:08:34.705340 21010 log.cc:824] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.707660 21010 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:34.707948 21010 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Bootstrap complete.
I20260808 21:08:34.708930 21010 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260808 21:08:34.709919 21010 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.710176 21010 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a523e37cfd147828101c003af349c1c, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.710361 21010 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.710632 21010 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:08:34.712175 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 20956
I20260808 21:08:34.712473 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:38427
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=40765
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:34.810727 21127 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:08:34.702805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:40393)
0808 21:08:34.703003 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.703054 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:34.703714 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:34.810537 (+106823us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
W20260808 21:08:34.811064 21125 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:37635 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:37635: Client connection negotiation failed: client connection to 127.4.177.126:37635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 21:08:34.811547 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.811381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:40727)
0808 21:08:34.811468 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.811526 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 21:08:34.811925 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.811840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:59769)
0808 21:08:34.811886 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.811909 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260808 21:08:34.812078 21125 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:08:34.829943 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.830139 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.830163 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.831347 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.831388 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.831437 21130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:34.832885 21130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=40765
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.833122 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.833348 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.838186 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.838203 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.838276 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.838366 21130 server_base.cc:1034] running on GCE node
I20260808 21:08:34.838708 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.838956 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.840121 21130 hybrid_clock.cc:648] HybridClock initialized: now 1786223314840083 us; error 68 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.842558 21130 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:34.843081 21130 webserver.cc:460] Webserver started at http://127.4.177.66:40765/ using document root <none> and password file <none>
I20260808 21:08:34.843343 21130 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.843412 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.844681 21130 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.845405 21145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.845563 21130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:34.845700 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "386b2a0f586f4a819f071d48b2cdd234"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "18ab915c0767e65ac0d53a5d8589a835"
server_key_iv: "efdbecadcde0997f3c5a711f6c29a7ed"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.846022 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.870730 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:34.873273 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:34.873451 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:34.873708 21130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:34.874179 21152 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 21:08:34.875727 21130 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 21:08:34.875797 21130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 21:08:34.875845 21130 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 21:08:34.876602 21130 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 21:08:34.876677 21130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 21:08:34.876751 21152 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Bootstrap starting.
I20260808 21:08:34.878309 21152 log.cc:824] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:34.880796 21152 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:34.881094 21152 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Bootstrap complete.
I20260808 21:08:34.881276 21152 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 21:08:34.882598 21152 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.882854 21152 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386b2a0f586f4a819f071d48b2cdd234, State: Initialized, Role: FOLLOWER
I20260808 21:08:34.883029 21152 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:34.883390 21152 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:08:34.884150 21130 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38427
I20260808 21:08:34.884203 21271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38427 every 8 connection(s)
I20260808 21:08:34.884500 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:08:34.885404 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.885076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:55067)
0808 21:08:34.885222 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.885288 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
W20260808 21:08:34.885588 21274 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:37635 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:37635: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
I20260808 21:08:34.885875 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.885774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:45655)
0808 21:08:34.885821 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.885847 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 21:08:34.886154 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:34.886063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:55397)
0808 21:08:34.886105 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:34.886135 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:08:34.886262 21274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 21:08:34.889382 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21130
I20260808 21:08:34.889621 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:40513
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:37635
--webserver_port=45123
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:44831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:34.970661 21278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:34.970853 21278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:34.970877 21278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:34.972050 21278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:34.972093 21278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:34.972121 21278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:34.973577 21278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:40513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=45123
--tserver_master_addrs=127.4.177.126:37635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:34.973779 21278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:34.974004 21278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:34.979115 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.979110 21286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:34.979287 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:34.979270 21278 server_base.cc:1034] running on GCE node
I20260808 21:08:34.979570 21278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:34.979777 21278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:34.980960 21278 hybrid_clock.cc:648] HybridClock initialized: now 1786223314980949 us; error 42 us; skew 500 ppm
Aug 08 21:08:34 dist-test-slave-chsk krb5kdc[19740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:34.983474 21278 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:34.983892 21278 webserver.cc:460] Webserver started at http://127.4.177.67:45123/ using document root <none> and password file <none>
I20260808 21:08:34.984143 21278 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:34.984213 21278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:34.985518 21278 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:08:34.986102 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.986286 21278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:34.986404 21278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "524e7f3f517546b18fed2e628283fa39"
format_stamp: "Formatted at 2026-08-08 21:08:33 on dist-test-slave-chsk"
server_key: "7b848847d5deaaca3796b76ebbd75cc8"
server_key_iv: "e9ebeb7e3ba249af4b7310065ddcdde8"
server_key_version: "encryptionkey@0"
I20260808 21:08:34.986737 21278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:34.998926 21278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:35.000900 21278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:35.001078 21278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:35.001314 21278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:35.001626 21278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:35.001683 21278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:35.001726 21278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:35.001755 21278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:35.007694 21278 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:40513
I20260808 21:08:35.007751 21413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:40513 every 8 connection(s)
I20260808 21:08:35.008122 21278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:08:35.009161 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:35.008814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:56869)
0808 21:08:35.008957 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:35.009025 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260808 21:08:35.009577 21414 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:37635 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:37635: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
I20260808 21:08:35.009846 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:35.009757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:56329)
0808 21:08:35.009802 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:35.009828 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260808 21:08:35.010183 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:35.010109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:55403)
0808 21:08:35.010147 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:35.010168 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260808 21:08:35.010298 21414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 21:08:35.015246 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21278
I20260808 21:08:35.812876 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:35.812623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:42211)
0808 21:08:35.812768 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:35.812810 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260808 21:08:35.886868 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:35.886623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:59159)
0808 21:08:35.886772 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:35.886815 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260808 21:08:36.010838 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.010613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:48941)
0808 21:08:36.010749 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.010786 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260808 21:08:36.030815 21421 raft_consensus.cc:482] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:36.030941 21421 raft_consensus.cc:504] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:36.031421 21421 leader_election.cc:288] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427), 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513)
Aug 08 21:08:36 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.67@KRBTEST.COM
Aug 08 21:08:36 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.66@KRBTEST.COM
I20260808 21:08:36.041671 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.031702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:41969 (local address 127.4.177.66:38427)
0808 21:08:36.031919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:36.031927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:36.031953 (+ 26us) server_negotiation.cc:407] Connection header received
0808 21:08:36.032087 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:36.032089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:36.032154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:36.032295 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:36.032965 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.034239 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035001 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.035469 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035550 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.038308 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:36.038312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:36.038312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:36.038324 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:36.040124 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040488 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.040490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.040491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.040581 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040818 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.040820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.040820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.041106 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:36.041173 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:36.041411 (+ 238us) server_negotiation.cc:299] Negotiation successful
0808 21:08:36.041550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
I20260808 21:08:36.041671 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.031734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:44863 (local address 127.4.177.67:40513)
0808 21:08:36.031913 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:36.031927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:36.031958 (+ 31us) server_negotiation.cc:407] Connection header received
0808 21:08:36.032543 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:36.032546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:36.032583 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:36.032717 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:36.033093 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.033937 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035044 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.035520 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035648 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.037933 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:36.037938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:36.037938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:36.037951 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:36.039753 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040160 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.040164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.040164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.040258 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040609 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.040610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.040610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.040963 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:36.041049 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:36.041406 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 21:08:36.041549 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260808 21:08:36.041699 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.031615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:40513 (local address 127.4.177.65:44863)
0808 21:08:36.031717 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.031735 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:36.031785 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:36.032713 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:36.032714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:36.032917 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:36.032983 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.032986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.034042 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.034043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.034918 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.034921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.035622 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.035686 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.036353 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:36.036357 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:36.037819 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:36.039844 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.039851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.039853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.040075 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.040338 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.040339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.040446 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.041039 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:36.041045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:36.041182 (+ 137us) client_negotiation.cc:769] Sending connection context
0808 21:08:36.041303 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 21:08:36.041512 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":28,"thread_start_us":58,"threads_started":1}
I20260808 21:08:36.042414 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.031592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38427 (local address 127.4.177.65:41969)
0808 21:08:36.031835 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.031848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:36.031912 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:36.032364 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:36.032366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:36.032656 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:36.032849 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.032854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.034330 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.034331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.034915 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.034917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.035557 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.035559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.035686 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.036309 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:36.036317 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:36.038182 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:36.040232 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.040234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.040406 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.040675 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.040676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.040677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.040735 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.041181 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:36.041182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:36.041216 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 21:08:36.041303 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 21:08:36.041569 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":124}
I20260808 21:08:36.042847 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "8a523e37cfd147828101c003af349c1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234" is_pre_election: true
W20260808 21:08:36.042868 21007 leader_election.cc:341] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513): Not found: Tablet not found: b670cddbb9f64485977e32bf411235d6
I20260808 21:08:36.043025 21223 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a523e37cfd147828101c003af349c1c in term 1.
I20260808 21:08:36.043313 21005 leader_election.cc:302] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 386b2a0f586f4a819f071d48b2cdd234, 8a523e37cfd147828101c003af349c1c; no voters: 524e7f3f517546b18fed2e628283fa39
I20260808 21:08:36.043475 21421 raft_consensus.cc:2781] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 21:08:36.043555 21421 raft_consensus.cc:482] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:36.043574 21421 raft_consensus.cc:3037] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:08:36.044713 21421 raft_consensus.cc:504] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:36.044872 21421 leader_election.cc:288] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 election: Requested vote from peers 386b2a0f586f4a819f071d48b2cdd234 (127.4.177.66:38427), 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513)
W20260808 21:08:36.045163 21007 leader_election.cc:341] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513): Not found: Tablet not found: b670cddbb9f64485977e32bf411235d6
I20260808 21:08:36.045148 21223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b670cddbb9f64485977e32bf411235d6" candidate_uuid: "8a523e37cfd147828101c003af349c1c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "386b2a0f586f4a819f071d48b2cdd234"
I20260808 21:08:36.045262 21223 raft_consensus.cc:3037] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:08:36.046382 21223 raft_consensus.cc:2454] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a523e37cfd147828101c003af349c1c in term 2.
I20260808 21:08:36.046710 21005 leader_election.cc:302] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 386b2a0f586f4a819f071d48b2cdd234, 8a523e37cfd147828101c003af349c1c; no voters: 524e7f3f517546b18fed2e628283fa39
I20260808 21:08:36.046805 21421 raft_consensus.cc:2781] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:08:36.046937 21421 raft_consensus.cc:686] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [term 2 LEADER]: Becoming Leader. State: Replica: 8a523e37cfd147828101c003af349c1c, State: Running, Role: LEADER
I20260808 21:08:36.047034 21421 consensus_queue.cc:237] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:36.047333 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.047212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:60613)
0808 21:08:36.047261 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.047299 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 21:08:36.428622 21007 consensus_peers.cc:487] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c -> Peer 524e7f3f517546b18fed2e628283fa39 (127.4.177.67:40513): Couldn't send request to peer 524e7f3f517546b18fed2e628283fa39. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b670cddbb9f64485977e32bf411235d6. This is attempt 1: this message will repeat every 5th retry.
I20260808 21:08:36.631831 21223 raft_consensus.cc:1261] T b670cddbb9f64485977e32bf411235d6 P 386b2a0f586f4a819f071d48b2cdd234 [term 2 FOLLOWER]: Refusing update from remote peer 8a523e37cfd147828101c003af349c1c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 21:08:36.632233 21421 consensus_queue.cc:1035] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 21:08:36.633148 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.632994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:59931)
0808 21:08:36.633078 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.633113 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260808 21:08:36.634009 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.633839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:48063)
0808 21:08:36.633933 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.633966 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
I20260808 21:08:36.634155 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.634063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:45523)
0808 21:08:36.634105 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.634128 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260808 21:08:36.879798 21436 ts_tablet_manager.cc:927] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Initiating tablet copy from peer 8a523e37cfd147828101c003af349c1c (127.4.177.65:35989)
I20260808 21:08:36.880049 21436 tablet_copy_client.cc:316] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: tablet copy: Beginning tablet copy session from remote peer at address 127.4.177.65:35989
Aug 08 21:08:36 dist-test-slave-chsk krb5kdc[19740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:08:36.888139 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.880430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:43627 (local address 127.4.177.65:35989)
0808 21:08:36.880614 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:36.880619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:36.880632 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:08:36.880684 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:36.880687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:36.880709 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:36.880785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:36.881314 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.881991 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.882804 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.883099 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.883210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.885540 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:36.885547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:36.885547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:36.885554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:36.886947 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.887437 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.887439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.887440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.887499 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:36.887685 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:36.887685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:36.887686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:36.887816 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:36.887878 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:36.888007 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 21:08:36.888071 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260808 21:08:36.888178 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.880316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35989 (local address 127.4.177.67:43627)
0808 21:08:36.880499 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.880536 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:36.880619 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:36.880803 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:36.880809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:36.881144 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:36.881215 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.881217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.882098 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.882099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.882699 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:36.882701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:36.883211 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:36.883212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:36.883315 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:36.883947 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:36.883952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:36.885454 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:36.887076 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.887085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.887088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.887364 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.887570 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:36.887570 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:36.887571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:36.887629 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:36.887872 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:36.887877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:36.887923 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 21:08:36.887986 (+ 63us) client_negotiation.cc:240] Negotiation successful
0808 21:08:36.888073 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
I20260808 21:08:36.888583 21097 tablet_copy_service.cc:140] P 8a523e37cfd147828101c003af349c1c: Received BeginTabletCopySession request for tablet b670cddbb9f64485977e32bf411235d6 from peer 524e7f3f517546b18fed2e628283fa39 ({username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:43627)
I20260808 21:08:36.888645 21097 tablet_copy_service.cc:161] P 8a523e37cfd147828101c003af349c1c: Beginning new tablet copy session on tablet b670cddbb9f64485977e32bf411235d6 from peer 524e7f3f517546b18fed2e628283fa39 at {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:43627: session id = 524e7f3f517546b18fed2e628283fa39-b670cddbb9f64485977e32bf411235d6
I20260808 21:08:36.889070 21097 tablet_copy_source_session.cc:205] T b670cddbb9f64485977e32bf411235d6 P 8a523e37cfd147828101c003af349c1c: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 21:08:36.889937 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b670cddbb9f64485977e32bf411235d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:36.892617 21436 tablet_copy_client.cc:791] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: tablet copy: Starting download of 0 data blocks...
I20260808 21:08:36.892807 21436 tablet_copy_client.cc:655] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: tablet copy: Starting download of 1 WAL segments...
I20260808 21:08:36.893543 21436 tablet_copy_client.cc:531] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 21:08:36.894156 21436 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Bootstrap starting.
I20260808 21:08:36.895675 21436 log.cc:824] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:36.898123 21436 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:08:36.898357 21436 tablet_bootstrap.cc:492] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Bootstrap complete.
I20260808 21:08:36.898499 21436 ts_tablet_manager.cc:1397] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260808 21:08:36.899515 21436 raft_consensus.cc:348] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:36.899683 21436 raft_consensus.cc:729] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524e7f3f517546b18fed2e628283fa39, State: Initialized, Role: FOLLOWER
I20260808 21:08:36.899850 21436 consensus_queue.cc:260] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "386b2a0f586f4a819f071d48b2cdd234" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38427 } } peers { permanent_uuid: "524e7f3f517546b18fed2e628283fa39" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 40513 } } peers { permanent_uuid: "8a523e37cfd147828101c003af349c1c" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35989 } }
I20260808 21:08:36.900107 21436 ts_tablet_manager.cc:1428] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260808 21:08:36.900422 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.900332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:35061)
0808 21:08:36.900364 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.900397 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260808 21:08:36.900403 21097 tablet_copy_service.cc:342] P 8a523e37cfd147828101c003af349c1c: Request end of tablet copy session 524e7f3f517546b18fed2e628283fa39-b670cddbb9f64485977e32bf411235d6 received from {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:43627
I20260808 21:08:36.900476 21097 tablet_copy_service.cc:433] P 8a523e37cfd147828101c003af349c1c: ending tablet copy session 524e7f3f517546b18fed2e628283fa39-b670cddbb9f64485977e32bf411235d6 on tablet b670cddbb9f64485977e32bf411235d6 with peer 524e7f3f517546b18fed2e628283fa39
I20260808 21:08:36.900781 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:36.900644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:34003)
0808 21:08:36.900743 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:36.900764 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 21:08:37.346660 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:37.346433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:43765)
0808 21:08:37.346555 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:37.346596 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260808 21:08:37.346660 21366 raft_consensus.cc:1203] T b670cddbb9f64485977e32bf411235d6 P 524e7f3f517546b18fed2e628283fa39 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260808 21:08:37.347224 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:37.347070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:53935)
0808 21:08:37.347158 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:37.347192 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 21:08:37.634917 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:37.634704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:41717)
0808 21:08:37.634824 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:37.634869 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260808 21:08:38.347677 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:38.347337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:43065)
0808 21:08:38.347550 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:38.347623 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260808 21:08:38.348006 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:38.347775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:47575)
0808 21:08:38.347912 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:38.347950 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260808 21:08:38.635658 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:38.635451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:46661)
0808 21:08:38.635570 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:38.635607 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260808 21:08:39.348435 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:39.348152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:47249)
0808 21:08:39.348317 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:39.348368 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260808 21:08:39.348697 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:39.348489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:58927)
0808 21:08:39.348622 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:39.348658 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260808 21:08:39.636261 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:39.636058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:45791)
0808 21:08:39.636179 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:39.636215 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260808 21:08:40.349301 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:40.349018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:58217)
0808 21:08:40.349196 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:40.349237 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260808 21:08:40.349452 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:40.349232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:50447)
0808 21:08:40.349359 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:40.349397 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260808 21:08:40.349689 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:40.349576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:55509)
0808 21:08:40.349629 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:40.349659 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 21:08:40.637006 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:40.636770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:60329)
0808 21:08:40.636920 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:40.636960 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260808 21:08:41.350271 21455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:41.350047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:56107)
0808 21:08:41.350178 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:41.350222 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260808 21:08:41.350364 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:41.350122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:35817)
0808 21:08:41.350269 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:41.350301 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260808 21:08:41.637743 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:41.637479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:45889)
0808 21:08:41.637657 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:41.637695 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
I20260808 21:08:42.350946 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:42.350725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:56019)
0808 21:08:42.350863 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:42.350898 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260808 21:08:42.350955 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:42.350727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:51867)
0808 21:08:42.350885 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:42.350914 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":79,"threads_started":1}
I20260808 21:08:42.638525 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:42.638265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:38421)
0808 21:08:42.638439 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:42.638478 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260808 21:08:43.351658 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:43.351402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:58875)
0808 21:08:43.351556 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:43.351605 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260808 21:08:43.351675 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:43.351446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:58801)
0808 21:08:43.351595 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:43.351630 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260808 21:08:43.639268 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:43.639010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:33679)
0808 21:08:43.639179 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:43.639213 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260808 21:08:44.352454 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:44.352198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:36757)
0808 21:08:44.352349 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:44.352395 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260808 21:08:44.352460 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:44.352211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:55229)
0808 21:08:44.352378 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:44.352417 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260808 21:08:44.640038 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:44.639736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:51583)
0808 21:08:44.639931 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:44.639984 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260808 21:08:45.353212 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:45.352988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:51161)
0808 21:08:45.353128 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:45.353162 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260808 21:08:45.353236 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:45.352988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:56317)
0808 21:08:45.353162 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:45.353194 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
I20260808 21:08:45.640825 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:45.640561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:44725)
0808 21:08:45.640726 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:45.640773 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260808 21:08:46.353889 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:46.353669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:35365)
0808 21:08:46.353808 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:46.353842 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260808 21:08:46.353915 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:46.353712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:39595)
0808 21:08:46.353844 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:46.353875 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260808 21:08:46.641559 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:46.641338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:44601)
0808 21:08:46.641472 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:46.641510 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260808 21:08:47.354599 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:47.354361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:46183)
0808 21:08:47.354501 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:47.354537 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260808 21:08:47.354826 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:47.354410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:45619)
0808 21:08:47.354713 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:47.354769 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260808 21:08:47.642390 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:47.642146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:56857)
0808 21:08:47.642309 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:47.642343 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
I20260808 21:08:48.355335 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:48.355051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:55159)
0808 21:08:48.355234 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:48.355270 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260808 21:08:48.355536 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:48.355320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:53915)
0808 21:08:48.355441 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:48.355481 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260808 21:08:48.643042 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:48.642827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:60709)
0808 21:08:48.642952 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:48.642993 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260808 21:08:49.356084 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:49.355820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.65:36159)
0808 21:08:49.355991 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:49.356035 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260808 21:08:49.356283 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:49.356025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.67:49157)
0808 21:08:49.356187 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:49.356228 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260808 21:08:49.643788 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:49.643547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:37635 (local address 127.4.177.66:35495)
0808 21:08:49.643688 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:49.643738 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:37635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260808 21:08:50.019131 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 20956
I20260808 21:08:50.025686 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21130
I20260808 21:08:50.032137 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21278
2026-08-08T21:08:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18365 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:08:50 dist-test-slave-chsk krb5kdc[21485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:08:50 dist-test-slave-chsk krb5kdc[21485](info): set up 2 sockets
Aug 08 21:08:50 dist-test-slave-chsk krb5kdc[21485](info): commencing operation
krb5kdc: starting...
W20260808 21:08:51.107865 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:08:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:08:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:08:51.228243 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:51.309793 21498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:51.310000 21498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:51.310022 21498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:51.311251 21498 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 21:08:51.311291 21498 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 21:08:51.311304 21498 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 21:08:51.311316 21498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:08:51.311357 21498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:51.311378 21498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:51.311399 21498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:08:51.311425 21498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:08:51.312980 21498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:51.313238 21498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:51.313479 21498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:51.318320 21504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.318295 21506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.318295 21503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:51.318740 21498 server_base.cc:1034] running on GCE node
I20260808 21:08:51.318974 21498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:51.319270 21498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:51.320423 21498 hybrid_clock.cc:648] HybridClock initialized: now 1786223331320405 us; error 44 us; skew 500 ppm
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:51.322966 21498 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:08:51.323495 21498 webserver.cc:460] Webserver started at http://127.4.177.126:43623/ using document root <none> and password file <none>
I20260808 21:08:51.323755 21498 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:51.323822 21498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:51.323966 21498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:51.325027 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6db95696da904e95aca51d5b49536163"
server_key_iv: "2ba87ec87847b6e7dc9fb785bd863719"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.325403 21498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6db95696da904e95aca51d5b49536163"
server_key_iv: "2ba87ec87847b6e7dc9fb785bd863719"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.326872 21498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:08:51.327668 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.327853 21498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.327948 21498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6db95696da904e95aca51d5b49536163"
server_key_iv: "2ba87ec87847b6e7dc9fb785bd863719"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.328028 21498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:51.341401 21498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:51.343266 21498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:51.343427 21498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:51.347730 21498 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:41229
I20260808 21:08:51.347769 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:41229 every 8 connection(s)
I20260808 21:08:51.348114 21498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:08:51.348986 21572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:51.351316 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap starting.
I20260808 21:08:51.351912 21572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:51.352221 21572 log.cc:824] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:51.352743 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21498
I20260808 21:08:51.352861 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:08:51.353020 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: No bootstrap required, opened a new log
I20260808 21:08:51.353093 4805 external_mini_cluster.cc:1428] Setting key 47937cbcf0ba64bf868f377163794b49
I20260808 21:08:51.354943 21572 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:08:51.355101 21572 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:51.355175 21572 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Initialized, Role: FOLLOWER
I20260808 21:08:51.355356 21572 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:08:51.355455 21572 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:08:51.355500 21572 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:08:51.355548 21572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:51.356081 21572 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:08:51.356208 21572 leader_election.cc:302] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb88b3d0d0146f49f8ffff4f16a36e6; no voters:
I20260808 21:08:51.356387 21572 leader_election.cc:288] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:08:51.356432 21577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:51.356608 21577 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 LEADER]: Becoming Leader. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Running, Role: LEADER
I20260808 21:08:51.356655 21572 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:08:51.356776 21577 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:08:51.357100 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:08:51.357177 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:51.357434 21584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:08:51.358086 21584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:08:51.358484 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb88b3d0d0146f49f8ffff4f16a36e6. Latest consensus state: current_term: 1 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:08:51.358657 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:08:51.360173 21584 catalog_manager.cc:1347] Generated new cluster ID: 94b53d1de40545159925171969f30f50
I20260808 21:08:51.360237 21584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:51.364593 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.353589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.4.177.126:41229)
0808 21:08:51.353804 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.353813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.353847 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:08:51.354285 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.354291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.354335 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.354480 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:51.355118 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.356568 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.357690 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.358458 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.358930 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.361331 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:51.361334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:51.361335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:51.361346 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:51.363062 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.363467 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.363470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.363471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.363560 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.363777 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.363778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.363778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.364009 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:51.364182 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.364263 (+ 81us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.364402 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260808 21:08:51.375443 21584 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:08:51.376134 21584 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:08:51.381158 21584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Generated new TSK 0
I20260808 21:08:51.381325 21584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:08:51.414317 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:51.496598 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:51.496843 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:51.496882 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:51.498025 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:51.498100 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:51.498147 21599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:08:51.499676 21599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:51.499951 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:51.500193 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:51.505324 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:51.505369 21599 server_base.cc:1034] running on GCE node
W20260808 21:08:51.505316 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.505316 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:51.505793 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:51.506024 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:51.507202 21599 hybrid_clock.cc:648] HybridClock initialized: now 1786223331507187 us; error 29 us; skew 500 ppm
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:51.509755 21599 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:08:51.510385 21599 webserver.cc:460] Webserver started at http://127.4.177.65:40523/ using document root <none> and password file <none>
I20260808 21:08:51.510634 21599 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:51.510686 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:51.510818 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:51.511947 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "de7463732182455595be7984203f6421"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "cc02943149e664d4bd41f6a70b5d413b"
server_key_iv: "c11a62e8a113a0ce208369624534c68a"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.512370 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "de7463732182455595be7984203f6421"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "cc02943149e664d4bd41f6a70b5d413b"
server_key_iv: "c11a62e8a113a0ce208369624534c68a"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.513731 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:08:51.514443 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.514654 21599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:08:51.514761 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "de7463732182455595be7984203f6421"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "cc02943149e664d4bd41f6a70b5d413b"
server_key_iv: "c11a62e8a113a0ce208369624534c68a"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.514848 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:51.524154 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:51.525874 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:51.526015 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:51.526250 21599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:51.526561 21599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:51.526594 21599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.526614 21599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:51.526659 21599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.532763 21599 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35181
I20260808 21:08:51.532838 21738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35181 every 8 connection(s)
I20260808 21:08:51.533143 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:51.540068 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21599
I20260808 21:08:51.540460 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:08:51.540750 4805 external_mini_cluster.cc:1428] Setting key e628be1b63cc4efe976bdc8d21776b11
I20260808 21:08:51.542737 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.533870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:55815 (local address 127.4.177.126:41229)
0808 21:08:51.533929 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.533931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.534118 (+ 187us) server_negotiation.cc:407] Connection header received
0808 21:08:51.534585 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.534586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.534596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.534647 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:51.535606 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.536108 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.537030 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.537221 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.537272 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.539435 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:51.539437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:51.539437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:51.539443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:51.541062 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.541670 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.541671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.541671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.541736 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.541974 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.541975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.541975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.542132 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:51.542206 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.542505 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.542600 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:08:51.542753 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.533852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:55815)
0808 21:08:51.534012 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.534051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.534501 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.534698 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.534701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.535335 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:51.535526 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.535531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.536191 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.536192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.536960 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.536962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.537312 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.537313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.537419 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.537914 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:51.537922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:51.539353 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:51.541334 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.541342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.541344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.541596 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.541810 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.541811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.541811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.541912 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.542231 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:51.542238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:51.542365 (+ 127us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.542465 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.542600 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260808 21:08:51.543512 21739 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:08:51.543622 21739 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:51.543815 21739 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:08:51.544363 21533 ts_manager.cc:194] Registered new tserver with Master: de7463732182455595be7984203f6421 (127.4.177.65:35181)
I20260808 21:08:51.544960 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:55815
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:08:51.568151 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:51.649011 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:51.649221 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:51.649246 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:51.650389 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:51.650430 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:51.650457 21744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:08:51.651916 21744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:51.652168 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:51.652421 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:08:51.657347 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.657356 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.657341 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:51.657398 21744 server_base.cc:1034] running on GCE node
I20260808 21:08:51.657804 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:51.658061 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:51.659199 21744 hybrid_clock.cc:648] HybridClock initialized: now 1786223331659174 us; error 37 us; skew 500 ppm
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:51.661643 21744 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:08:51.662117 21744 webserver.cc:460] Webserver started at http://127.4.177.66:45039/ using document root <none> and password file <none>
I20260808 21:08:51.662358 21744 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:51.662428 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:51.662585 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:51.663704 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "d31185cc22764db3959addff7a74e7d8"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "06a5343fb6dc98663df1a30f64a4aa49"
server_key_iv: "cedb0e6830645bb842eba08869af2cfb"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.664065 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "d31185cc22764db3959addff7a74e7d8"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "06a5343fb6dc98663df1a30f64a4aa49"
server_key_iv: "cedb0e6830645bb842eba08869af2cfb"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.665369 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:08:51.666049 21760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.666229 21744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:51.666326 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "d31185cc22764db3959addff7a74e7d8"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "06a5343fb6dc98663df1a30f64a4aa49"
server_key_iv: "cedb0e6830645bb842eba08869af2cfb"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.666411 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:51.678753 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:51.681027 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:51.681187 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:51.681414 21744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:51.681711 21744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:51.681766 21744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.681808 21744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:51.681836 21744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.687516 21744 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37807
I20260808 21:08:51.687567 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37807 every 8 connection(s)
I20260808 21:08:51.687870 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:08:51.693341 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21744
I20260808 21:08:51.693450 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:51.693672 4805 external_mini_cluster.cc:1428] Setting key 2c8f1e159cf6b24c17db89254e8e8063
I20260808 21:08:51.696925 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.688583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:44221 (local address 127.4.177.126:41229)
0808 21:08:51.688649 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.688659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.688841 (+ 182us) server_negotiation.cc:407] Connection header received
0808 21:08:51.689311 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.689312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.689322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.689379 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:51.690232 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.690723 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.691701 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.691878 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.691929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.694091 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:51.694093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:51.694093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:51.694100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:51.695582 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.695982 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.695983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.695984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.696039 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.696301 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.696302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.696303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.696418 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:51.696493 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.696758 (+ 265us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.696837 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:08:51.697103 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.688554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:44221)
0808 21:08:51.688728 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.688769 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.689228 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.689428 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.689431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.689982 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:51.690157 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.690161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.690800 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.690801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.691632 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.691634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.691947 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.691948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.692045 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.692449 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:51.692456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:51.693989 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:51.695678 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.695685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.695687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.695905 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.696112 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.696113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.696113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.696231 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.696485 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:51.696491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:51.696623 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.696730 (+ 107us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.696878 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260808 21:08:51.697489 21882 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:08:51.697595 21882 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:51.697762 21882 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:08:51.698127 21533 ts_manager.cc:194] Registered new tserver with Master: d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
I20260808 21:08:51.698488 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:44221
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:08:51.719408 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:08:51.803447 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:08:51.803684 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:08:51.803722 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:08:51.804833 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:08:51.804904 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:08:51.804951 21887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:08:51.806401 21887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:08:51.806665 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:08:51.806892 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:08:51.811982 21887 server_base.cc:1034] running on GCE node
W20260808 21:08:51.811992 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.811985 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:08:51.811985 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:08:51.812497 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:08:51.812700 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:08:51.813860 21887 hybrid_clock.cc:648] HybridClock initialized: now 1786223331813843 us; error 35 us; skew 500 ppm
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:08:51.816388 21887 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:08:51.816862 21887 webserver.cc:460] Webserver started at http://127.4.177.67:43317/ using document root <none> and password file <none>
I20260808 21:08:51.817097 21887 fs_manager.cc:359] Metadata directory not provided
I20260808 21:08:51.817164 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:08:51.817309 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:08:51.818389 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "ab803f0628174e718313e8a18d3003a3"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6d046b38c232dd8d4be3280af4f5c746"
server_key_iv: "0b173db98e99671da9a0633f77294bc3"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.818822 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "ab803f0628174e718313e8a18d3003a3"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6d046b38c232dd8d4be3280af4f5c746"
server_key_iv: "0b173db98e99671da9a0633f77294bc3"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.820225 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:08:51.820981 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.821152 21887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:08:51.821285 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "ab803f0628174e718313e8a18d3003a3"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6d046b38c232dd8d4be3280af4f5c746"
server_key_iv: "0b173db98e99671da9a0633f77294bc3"
server_key_version: "encryptionkey@0"
I20260808 21:08:51.821362 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:08:51.840883 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:08:51.843436 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:08:51.843611 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:08:51.843884 21887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:08:51.844205 21887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:08:51.844264 21887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.844308 21887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:08:51.844334 21887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:08:51.850319 21887 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46139
I20260808 21:08:51.850378 22023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46139 every 8 connection(s)
I20260808 21:08:51.850726 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:08:51.854161 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 21887
I20260808 21:08:51.854277 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:08:51.854480 4805 external_mini_cluster.cc:1428] Setting key 472e4112e818f7a761c90220dedfed6c
Aug 08 21:08:51 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:08:51.860836 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.851664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:54747 (local address 127.4.177.126:41229)
0808 21:08:51.851739 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.851741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.851911 (+ 170us) server_negotiation.cc:407] Connection header received
0808 21:08:51.852371 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.852372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.852382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.852441 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:51.853440 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.854093 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.855596 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.855790 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.855848 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.858286 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:51.858287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:51.858287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:51.858294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:51.859628 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.860010 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.860011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.860012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.860079 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.860315 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.860316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.860316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.860432 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:51.860486 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.860707 (+ 221us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.860771 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 21:08:51.861066 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.851609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:54747)
0808 21:08:51.851775 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.851816 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.852288 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.852499 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.852502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.853163 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:08:51.853357 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.853362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.854187 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.854188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.855495 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.855503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.855863 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.855864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.855970 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.856583 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:08:51.856591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:08:51.858201 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:08:51.859701 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.859707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.859709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.859925 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.860147 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:08:51.860148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:08:51.860148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:08:51.860257 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:08:51.860489 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:08:51.860493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:08:51.860602 (+ 109us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.860685 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.860806 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260808 21:08:51.861557 22024 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:08:51.861654 22024 heartbeater.cc:464] Registering TS with master...
I20260808 21:08:51.861811 22024 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:08:51.862215 21533 ts_manager.cc:194] Registered new tserver with Master: ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139)
I20260808 21:08:51.862578 21533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:54747
I20260808 21:08:51.866312 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:08:51.873430 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.867686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.4.177.126:41229)
0808 21:08:51.867747 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.867749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.867883 (+ 134us) server_negotiation.cc:407] Connection header received
0808 21:08:51.867917 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.867917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.867926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.867978 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:08:51.868521 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.869049 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.869755 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.869931 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.869982 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.871296 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:08:51.871297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:08:51.871298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:08:51.871304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:08:51.872476 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.872778 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.872779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.872779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.872832 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:08:51.872963 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:08:51.872963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:08:51.872964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:08:51.873099 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:08:51.873174 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.873270 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.873331 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:08:51.875201 21533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:08:51.876797 21533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:08:51.883524 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.879632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.4.177.65:35181)
0808 21:08:51.879833 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.879837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.879848 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:08:51.879887 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.879889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.879903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.879983 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:51.880184 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.881252 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.882589 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883072 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883214 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.883385 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.883429 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260808 21:08:51.883559 22036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.879247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37807 (local address 127.0.0.1:55318)
0808 21:08:51.879339 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.879350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.879410 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.880012 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.880013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.880016 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:51.880071 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.880073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.881101 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.881102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.882291 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.882294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883289 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.883361 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883366 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.883397 (+ 31us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.883444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":30,"thread_start_us":55,"threads_started":1}
I20260808 21:08:51.883664 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.878983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46139 (local address 127.0.0.1:57698)
0808 21:08:51.879119 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.879182 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.879247 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.879537 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.879541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.879544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:51.879617 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.879620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.880957 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.880958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.882478 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.882481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883449 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.883506 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883509 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.883570 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.883608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260808 21:08:51.883822 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.879459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.4.177.66:37807)
0808 21:08:51.879851 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.879857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.879866 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:08:51.879911 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.879913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.879930 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.880003 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:51.880126 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.880985 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.882396 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883181 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883603 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883669 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.883696 (+ 27us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.883746 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":88,"threads_started":1}
I20260808 21:08:51.883888 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.879086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.4.177.67:46139)
0808 21:08:51.879284 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:51.879289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:51.879301 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:08:51.879343 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:51.879346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:51.879365 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:51.879445 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:51.879717 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.880846 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.882574 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883304 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883417 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:51.883758 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 21:08:51.883801 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":131,"threads_started":1}
I20260808 21:08:51.883549 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:51.879388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35181 (local address 127.0.0.1:58874)
0808 21:08:51.879658 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:51.879669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:51.879724 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:51.880063 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:51.880063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:51.880066 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:51.880106 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.880108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.881356 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.881357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.882496 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:51.882498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:51.883199 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:51.883200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:51.883278 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:51.883293 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 21:08:51.883370 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 21:08:51.883429 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":42,"threads_started":1}
I20260808 21:08:51.884605 21954 tablet_service.cc:1467] Processing CreateTablet for tablet fd979d3649f44a09973e467936a1e10e (DEFAULT_TABLE table=test-table [id=b2f53a6a2c864b0c956485b1954267d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:51.884721 21811 tablet_service.cc:1467] Processing CreateTablet for tablet fd979d3649f44a09973e467936a1e10e (DEFAULT_TABLE table=test-table [id=b2f53a6a2c864b0c956485b1954267d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:51.884994 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd979d3649f44a09973e467936a1e10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:51.884994 21954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd979d3649f44a09973e467936a1e10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:51.885072 21668 tablet_service.cc:1467] Processing CreateTablet for tablet fd979d3649f44a09973e467936a1e10e (DEFAULT_TABLE table=test-table [id=b2f53a6a2c864b0c956485b1954267d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:08:51.885419 21668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd979d3649f44a09973e467936a1e10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:08:51.887984 22040 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Bootstrap starting.
I20260808 21:08:51.888346 22041 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Bootstrap starting.
I20260808 21:08:51.888666 22042 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Bootstrap starting.
I20260808 21:08:51.888983 22040 tablet_bootstrap.cc:654] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:51.889050 22041 tablet_bootstrap.cc:654] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:51.889268 22042 tablet_bootstrap.cc:654] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Neither blocks nor log segments found. Creating new log.
I20260808 21:08:51.889642 22041 log.cc:824] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:51.889642 22040 log.cc:824] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:51.889839 22042 log.cc:824] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Log is configured to *not* fsync() on all Append() calls
I20260808 21:08:51.890489 22040 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: No bootstrap required, opened a new log
I20260808 21:08:51.890571 22040 ts_tablet_manager.cc:1397] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 21:08:51.890646 22041 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: No bootstrap required, opened a new log
I20260808 21:08:51.890666 22042 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: No bootstrap required, opened a new log
I20260808 21:08:51.890724 22041 ts_tablet_manager.cc:1397] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:51.890759 22042 ts_tablet_manager.cc:1397] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:08:51.892189 22041 raft_consensus.cc:348] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.892334 22041 raft_consensus.cc:374] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:51.892357 22041 raft_consensus.cc:729] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab803f0628174e718313e8a18d3003a3, State: Initialized, Role: FOLLOWER
I20260808 21:08:51.892524 22041 consensus_queue.cc:260] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.892617 22040 raft_consensus.cc:348] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.892817 22040 raft_consensus.cc:374] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:51.892838 22024 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
I20260808 21:08:51.892868 22040 raft_consensus.cc:729] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7463732182455595be7984203f6421, State: Initialized, Role: FOLLOWER
I20260808 21:08:51.892848 22041 ts_tablet_manager.cc:1428] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:08:51.892995 22040 consensus_queue.cc:260] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.893033 22042 raft_consensus.cc:348] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.893204 22042 raft_consensus.cc:374] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:08:51.893244 22042 raft_consensus.cc:729] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31185cc22764db3959addff7a74e7d8, State: Initialized, Role: FOLLOWER
I20260808 21:08:51.893282 22040 ts_tablet_manager.cc:1428] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:08:51.893352 22042 consensus_queue.cc:260] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:51.893605 21739 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
I20260808 21:08:51.893631 21882 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
I20260808 21:08:51.893625 22042 ts_tablet_manager.cc:1428] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260808 21:08:51.938441 21883 tablet.cc:2369] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:08:52.033866 21740 tablet.cc:2369] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:08:52.057727 22047 raft_consensus.cc:482] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:08:52.057890 22047 raft_consensus.cc:504] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:52.058271 22047 leader_election.cc:288] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
I20260808 21:08:52.061476 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.058554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:57011 (local address 127.4.177.67:46139)
0808 21:08:52.058615 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:52.058617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:52.058749 (+ 132us) server_negotiation.cc:407] Connection header received
0808 21:08:52.058810 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:52.058811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:52.058821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:52.058900 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:52.059072 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.059618 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.060742 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.061114 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.061203 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:52.061213 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:52.061366 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 21:08:52.061402 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 21:08:52.061522 22049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.058484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46139 (local address 127.4.177.65:57011)
0808 21:08:52.058651 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:52.058663 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:52.058745 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:52.058905 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:52.058907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:52.058909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:52.058988 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:52.058990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.059707 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.059708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:52.060639 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:52.060641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.061201 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.061202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:52.061272 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:52.061278 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 21:08:52.061357 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:08:52.061400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":38,"thread_start_us":122,"threads_started":1}
I20260808 21:08:52.061527 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.058827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:59443 (local address 127.4.177.66:37807)
0808 21:08:52.058888 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:52.058891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:52.058902 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:08:52.058951 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:52.058952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:52.058961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:52.059028 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:08:52.059238 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.059692 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.060773 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.061149 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.061257 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:52.061266 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:52.061420 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 21:08:52.061456 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 21:08:52.061550 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.058740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37807 (local address 127.4.177.65:59443)
0808 21:08:52.058846 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:52.058856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:08:52.058895 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:08:52.059044 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:08:52.059045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:08:52.059046 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:08:52.059148 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:52.059149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.059782 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.059783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:52.060673 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:08:52.060676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.061267 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.061269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:08:52.061347 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:52.061350 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:08:52.061416 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 21:08:52.061455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260808 21:08:52.061926 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd979d3649f44a09973e467936a1e10e" candidate_uuid: "de7463732182455595be7984203f6421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31185cc22764db3959addff7a74e7d8" is_pre_election: true
I20260808 21:08:52.061930 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd979d3649f44a09973e467936a1e10e" candidate_uuid: "de7463732182455595be7984203f6421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab803f0628174e718313e8a18d3003a3" is_pre_election: true
I20260808 21:08:52.062072 21834 raft_consensus.cc:2454] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7463732182455595be7984203f6421 in term 0.
I20260808 21:08:52.062093 21976 raft_consensus.cc:2454] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de7463732182455595be7984203f6421 in term 0.
I20260808 21:08:52.062291 21617 leader_election.cc:302] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d31185cc22764db3959addff7a74e7d8, de7463732182455595be7984203f6421; no voters:
I20260808 21:08:52.062422 22047 raft_consensus.cc:2781] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:08:52.062518 22047 raft_consensus.cc:482] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:08:52.062556 22047 raft_consensus.cc:3037] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:52.063400 22047 raft_consensus.cc:504] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:52.063552 22047 leader_election.cc:288] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [CANDIDATE]: Term 1 election: Requested vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
I20260808 21:08:52.063791 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd979d3649f44a09973e467936a1e10e" candidate_uuid: "de7463732182455595be7984203f6421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab803f0628174e718313e8a18d3003a3"
I20260808 21:08:52.063788 21834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd979d3649f44a09973e467936a1e10e" candidate_uuid: "de7463732182455595be7984203f6421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31185cc22764db3959addff7a74e7d8"
I20260808 21:08:52.063891 21976 raft_consensus.cc:3037] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:52.063892 21834 raft_consensus.cc:3037] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:08:52.064841 21834 raft_consensus.cc:2454] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7463732182455595be7984203f6421 in term 1.
I20260808 21:08:52.064841 21976 raft_consensus.cc:2454] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de7463732182455595be7984203f6421 in term 1.
I20260808 21:08:52.065061 21617 leader_election.cc:302] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab803f0628174e718313e8a18d3003a3, de7463732182455595be7984203f6421; no voters:
I20260808 21:08:52.065188 22047 raft_consensus.cc:2781] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:08:52.065374 22047 raft_consensus.cc:686] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 LEADER]: Becoming Leader. State: Replica: de7463732182455595be7984203f6421, State: Running, Role: LEADER
I20260808 21:08:52.065497 22047 consensus_queue.cc:237] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:08:52.066422 21531 catalog_manager.cc:5495] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 reported cstate change: term changed from 0 to 1, leader changed from <none> to de7463732182455595be7984203f6421 (127.4.177.65). New cstate: current_term: 1 leader_uuid: "de7463732182455595be7984203f6421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } health_report { overall_health: HEALTHY } } }
W20260808 21:08:52.101408 22026 tablet.cc:2369] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:08:52.109609 22039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.107370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.4.177.65:35181)
0808 21:08:52.107438 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0808 21:08:52.107440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:08:52.107450 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:08:52.107498 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:08:52.107498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:08:52.107518 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:08:52.107578 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:08:52.107718 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.108180 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.108937 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:08:52.109114 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:08:52.109162 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:08:52.109310 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:08:52.109411 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:08:52.109472 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:08:52.109501 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 21:08:52.109518 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 21:08:52.111539 21834 raft_consensus.cc:1261] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Refusing update from remote peer de7463732182455595be7984203f6421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:52.111550 21976 raft_consensus.cc:1261] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Refusing update from remote peer de7463732182455595be7984203f6421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:08:52.111941 22047 consensus_queue.cc:1035] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:52.112121 22052 consensus_queue.cc:1035] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:08:52.114709 22056 mvcc.cc:205] Tried to move back new op lower bound from 7316370768325881856 to 7316370768140935168. Current Snapshot: MvccSnapshot[applied={T|T < 7316370768325881856}]
I20260808 21:08:52.114753 22055 mvcc.cc:205] Tried to move back new op lower bound from 7316370768325881856 to 7316370768140935168. Current Snapshot: MvccSnapshot[applied={T|T < 7316370768325881856}]
I20260808 21:08:52.115103 22057 mvcc.cc:205] Tried to move back new op lower bound from 7316370768325881856 to 7316370768140935168. Current Snapshot: MvccSnapshot[applied={T|T < 7316370768325881856}]
I20260808 21:08:52.116803 21533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:52.116910 21533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:08:52.117676 21533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Sending DeleteTablet for 3 replicas of tablet fd979d3649f44a09973e467936a1e10e
I20260808 21:08:52.117967 21954 tablet_service.cc:1514] Processing DeleteTablet for tablet fd979d3649f44a09973e467936a1e10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:52 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:57698
I20260808 21:08:52.118014 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21599
I20260808 21:08:52.118048 21811 tablet_service.cc:1514] Processing DeleteTablet for tablet fd979d3649f44a09973e467936a1e10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:52 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:55318
I20260808 21:08:52.119509 22067 tablet_replica.cc:330] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: stopping tablet replica
I20260808 21:08:52.119647 22067 raft_consensus.cc:2229] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:52.119758 22067 raft_consensus.cc:2258] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:52.120191 22067 ts_tablet_manager.cc:1905] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:52.121433 22066 tablet_replica.cc:330] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: stopping tablet replica
I20260808 21:08:52.121560 22066 raft_consensus.cc:2229] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:08:52.121688 22066 raft_consensus.cc:2258] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:08:52.121941 22067 ts_tablet_manager.cc:1918] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:52.122097 22066 ts_tablet_manager.cc:1905] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:08:52.122113 22067 log.cc:1196] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/fd979d3649f44a09973e467936a1e10e
I20260808 21:08:52.122486 22067 ts_tablet_manager.cc:1939] T fd979d3649f44a09973e467936a1e10e P d31185cc22764db3959addff7a74e7d8: Deleting consensus metadata
I20260808 21:08:52.123116 21518 catalog_manager.cc:4841] TS d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): tablet fd979d3649f44a09973e467936a1e10e (table test-table [id=b2f53a6a2c864b0c956485b1954267d2]) successfully deleted
I20260808 21:08:52.123790 22066 ts_tablet_manager.cc:1918] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:08:52.123867 22066 log.cc:1196] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/fd979d3649f44a09973e467936a1e10e
I20260808 21:08:52.124203 22066 ts_tablet_manager.cc:1939] T fd979d3649f44a09973e467936a1e10e P ab803f0628174e718313e8a18d3003a3: Deleting consensus metadata
I20260808 21:08:52.124588 21518 catalog_manager.cc:4841] TS ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): tablet fd979d3649f44a09973e467936a1e10e (table test-table [id=b2f53a6a2c864b0c956485b1954267d2]) successfully deleted
W20260808 21:08:52.127753 21518 connection.cc:535] client connection to 127.4.177.65:35181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 21:08:52.127859 21518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 21:08:52.127964 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21744
I20260808 21:08:52.135538 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:08:52.135360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35181 (local address 127.0.0.1:58884)
0808 21:08:52.135431 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0808 21:08:52.135486 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260808 21:08:52.135968 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21887
W20260808 21:08:52.135982 21518 catalog_manager.cc:4584] TS de7463732182455595be7984203f6421 (127.4.177.65:35181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd979d3649f44a09973e467936a1e10e: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35181: connect: Connection refused (error 111)
I20260808 21:08:52.143989 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 21498
I20260808 21:09:12.152212 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--webserver_interface=127.4.177.126
--webserver_port=43623
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:12.236949 22069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:12.237176 22069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:12.237200 22069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:12.238373 22069 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 21:09:12.238417 22069 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 21:09:12.238431 22069 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 21:09:12.238442 22069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:12.238456 22069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:12.238469 22069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:12.238524 22069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:12.238540 22069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:12.240278 22069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:12.240548 22069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:12.240808 22069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:12.250142 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.250324 22069 server_base.cc:1034] running on GCE node
W20260808 21:09:12.250152 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:12.250161 22079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.250736 22069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:12.251106 22069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:12.252321 22069 hybrid_clock.cc:648] HybridClock initialized: now 1786223352252296 us; error 35 us; skew 500 ppm
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:12.260650 22069 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:09:12.261152 22069 webserver.cc:460] Webserver started at http://127.4.177.126:43623/ using document root <none> and password file <none>
I20260808 21:09:12.261412 22069 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:12.261504 22069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:12.263008 22069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.263780 22086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.263968 22069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.264083 22069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6db95696da904e95aca51d5b49536163"
server_key_iv: "2ba87ec87847b6e7dc9fb785bd863719"
server_key_version: "encryptionkey@0"
I20260808 21:09:12.264501 22069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:12.300711 22069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:12.303349 22069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:12.303522 22069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:12.307711 22069 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:41229
I20260808 21:09:12.307753 22140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:41229 every 8 connection(s)
I20260808 21:09:12.308190 22069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:12.308739 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22069
I20260808 21:09:12.309048 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:35181
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=40523
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:09:12.309880 22141 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:09:12.310567 22141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap starting.
I20260808 21:09:12.319700 22141 log.cc:824] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:12.321730 22141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:09:12.322073 22141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap complete.
I20260808 21:09:12.324481 22141 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:12.324745 22141 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Initialized, Role: FOLLOWER
I20260808 21:09:12.324955 22141 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:12.325057 22141 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:12.325117 22141 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:12.325169 22141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 1 FOLLOWER]: Advancing to term 2
I20260808 21:09:12.326205 22141 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:12.326330 22141 leader_election.cc:302] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb88b3d0d0146f49f8ffff4f16a36e6; no voters:
I20260808 21:09:12.326545 22141 leader_election.cc:288] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 21:09:12.326586 22145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 21:09:12.326709 22145 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 LEADER]: Becoming Leader. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Running, Role: LEADER
I20260808 21:09:12.326848 22141 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:12.326864 22145 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:12.327268 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb88b3d0d0146f49f8ffff4f16a36e6. Latest consensus state: current_term: 2 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:09:12.327273 22146 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:09:12.327633 22146 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:12.327467 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:12.327865 22154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:12.328778 22154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:12.328938 22154 catalog_manager.cc:1259] Loaded cluster ID: 94b53d1de40545159925171969f30f50
I20260808 21:09:12.329141 22154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:12.330297 22154 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:12.330466 22154 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Loaded TSK: 0
I20260808 21:09:12.342518 22154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Generated new TSK 1
I20260808 21:09:12.342628 22154 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 21:09:12.395742 22143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:12.395946 22143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:12.395968 22143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:12.397145 22143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:12.397190 22143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:12.397217 22143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:12.398694 22143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:35181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=40523
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:12.398938 22143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:12.399262 22143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:12.404321 22169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:12.404301 22170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.404469 22143 server_base.cc:1034] running on GCE node
W20260808 21:09:12.404305 22172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.404907 22143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:12.405221 22143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:12.406416 22143 hybrid_clock.cc:648] HybridClock initialized: now 1786223352406402 us; error 28 us; skew 500 ppm
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:12.409482 22143 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:09:12.409991 22143 webserver.cc:460] Webserver started at http://127.4.177.65:40523/ using document root <none> and password file <none>
I20260808 21:09:12.410251 22143 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:12.410337 22143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:12.411829 22143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:09:12.412593 22179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.412796 22143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.412911 22143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "de7463732182455595be7984203f6421"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "cc02943149e664d4bd41f6a70b5d413b"
server_key_iv: "c11a62e8a113a0ce208369624534c68a"
server_key_version: "encryptionkey@0"
I20260808 21:09:12.413280 22143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:12.441551 22143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:12.444124 22143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:12.444303 22143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:12.444567 22143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:12.445107 22186 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 21:09:12.446666 22143 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 21:09:12.446722 22143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 21:09:12.446750 22143 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 21:09:12.447520 22143 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 21:09:12.447567 22143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.447618 22186 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Bootstrap starting.
I20260808 21:09:12.455616 22143 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35181
I20260808 21:09:12.455693 22302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35181 every 8 connection(s)
I20260808 21:09:12.456254 22143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:12.459892 22186 log.cc:824] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:12.462435 22186 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:09:12.462792 22186 tablet_bootstrap.cc:492] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Bootstrap complete.
I20260808 21:09:12.464047 22186 ts_tablet_manager.cc:1397] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20260808 21:09:12.465332 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22143
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:12.465307 22186 raft_consensus.cc:348] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.465654 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:37807
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=45039
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:09:12.465662 22186 raft_consensus.cc:729] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7463732182455595be7984203f6421, State: Initialized, Role: FOLLOWER
I20260808 21:09:12.465849 22186 consensus_queue.cc:260] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.466187 22186 ts_tablet_manager.cc:1428] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 21:09:12.471087 22305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.456994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:51881)
0808 21:09:12.457191 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.457261 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.458007 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.459123 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.459128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.459572 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:12.459862 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.459870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.461420 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.461421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.462717 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.462721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.463412 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.463413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.463572 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.464148 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:12.464160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:12.466069 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:12.469373 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.469380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.469382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.469636 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.469938 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.469939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.469939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.470037 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.470514 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:12.470519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:12.470639 (+ 120us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.470722 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.470854 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260808 21:09:12.471264 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.457269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:51881 (local address 127.4.177.126:41229)
0808 21:09:12.458614 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.458624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.458657 (+ 33us) server_negotiation.cc:407] Connection header received
0808 21:09:12.458794 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.458801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.458870 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.459022 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.460014 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.461276 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.462813 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.463308 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.463393 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.466201 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.466205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.466206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.466220 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.469279 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.469728 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.469731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.469732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.469865 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.470105 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.470106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.470107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.470439 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.470543 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.470823 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.470995 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":86,"threads_started":1}
I20260808 21:09:12.471937 22303 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:09:12.472060 22303 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:12.472249 22303 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:09:12.473012 22104 ts_manager.cc:194] Registered new tserver with Master: de7463732182455595be7984203f6421 (127.4.177.65:35181)
I20260808 21:09:12.474031 22104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:51881
I20260808 21:09:12.474882 22303 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.65@KRBTEST.COM
I20260808 21:09:12.482805 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.473735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.4.177.65:35181)
0808 21:09:12.474181 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.474185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.474195 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:12.474232 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.474235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.474252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.474322 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.474991 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.476212 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.476958 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.477475 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.477561 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.479862 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.479870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.479870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.479876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.481549 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.481999 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.482001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.482002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.482068 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.482269 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.482270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.482270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.482407 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.482470 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.482656 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.482733 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":91,"threads_started":1}
I20260808 21:09:12.482868 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.473662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35181 (local address 127.0.0.1:44124)
0808 21:09:12.473926 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.473963 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.474035 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.474397 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.474403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.474813 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:12.474885 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.474888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.476300 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.476301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.476868 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.476870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.477569 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.477570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.477628 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.478158 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:12.478168 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:12.479774 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:12.481641 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.481648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.481650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.481917 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.482136 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.482137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.482137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.482207 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.482471 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:12.482479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:12.482535 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.482632 (+ 97us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.482760 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
I20260808 21:09:12.483589 22234 tablet_service.cc:1514] Processing DeleteTablet for tablet fd979d3649f44a09973e467936a1e10e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:08:52 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:44124
I20260808 21:09:12.484225 22312 tablet_replica.cc:330] stopping tablet replica
I20260808 21:09:12.484349 22312 raft_consensus.cc:2229] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:12.484427 22312 raft_consensus.cc:2258] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:12.484735 22312 ts_tablet_manager.cc:1905] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:12.487222 22312 ts_tablet_manager.cc:1918] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:12.487322 22312 log.cc:1196] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal/wals/fd979d3649f44a09973e467936a1e10e
I20260808 21:09:12.487589 22312 ts_tablet_manager.cc:1939] T fd979d3649f44a09973e467936a1e10e P de7463732182455595be7984203f6421: Deleting consensus metadata
I20260808 21:09:12.488137 22088 catalog_manager.cc:4841] TS de7463732182455595be7984203f6421 (127.4.177.65:35181): tablet fd979d3649f44a09973e467936a1e10e (table test-table [id=b2f53a6a2c864b0c956485b1954267d2]) successfully deleted
W20260808 21:09:12.554664 22309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:12.554924 22309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:12.554965 22309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:12.556198 22309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:12.556272 22309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:12.556315 22309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:12.557818 22309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:37807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=45039
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:12.558102 22309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:12.558351 22309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:12.563427 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:12.563433 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:12.563602 22318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.563737 22309 server_base.cc:1034] running on GCE node
I20260808 21:09:12.563968 22309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:12.564219 22309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:12.565403 22309 hybrid_clock.cc:648] HybridClock initialized: now 1786223352565372 us; error 45 us; skew 500 ppm
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:12.568455 22309 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:09:12.568977 22309 webserver.cc:460] Webserver started at http://127.4.177.66:45039/ using document root <none> and password file <none>
I20260808 21:09:12.569231 22309 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:12.569329 22309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:12.570786 22309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:09:12.571502 22329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.571681 22309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:12.571794 22309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "d31185cc22764db3959addff7a74e7d8"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "06a5343fb6dc98663df1a30f64a4aa49"
server_key_iv: "cedb0e6830645bb842eba08869af2cfb"
server_key_version: "encryptionkey@0"
I20260808 21:09:12.572182 22309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:12.589520 22309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:12.592087 22309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:12.592264 22309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:12.592535 22309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:12.592921 22309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:12.592983 22309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.593025 22309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:12.593061 22309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.599114 22309 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37807
I20260808 21:09:12.599198 22452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37807 every 8 connection(s)
I20260808 21:09:12.599524 22309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:12.603061 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22309
I20260808 21:09:12.603444 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:46139
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=43317
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:12.609694 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.600224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:56563)
0808 21:09:12.600412 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.600454 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.600958 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.601231 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.601235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.601815 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:12.602007 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.602061 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.602741 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.602742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.603823 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.603826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.604238 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.604239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.604375 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.604923 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:12.604933 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:12.606312 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:12.608259 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.608266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.608267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.608488 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.608745 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.608746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.608747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.608859 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.609181 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:12.609186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:12.609306 (+ 120us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.609391 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.609525 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260808 21:09:12.609694 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.600254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:56563 (local address 127.4.177.126:41229)
0808 21:09:12.600343 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.600345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.600552 (+ 207us) server_negotiation.cc:407] Connection header received
0808 21:09:12.601071 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.601073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.601088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.601156 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.602151 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.602660 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.603929 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.604143 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.604216 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.606404 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.606405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.606406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.606415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.608171 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.608581 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.608582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.608583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.608652 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.608933 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.608934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.608934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.609095 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.609187 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.609444 (+ 257us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.609553 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 21:09:12.610327 22453 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:09:12.610450 22453 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:12.610632 22453 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:09:12.611050 22104 ts_manager.cc:194] Registered new tserver with Master: d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
I20260808 21:09:12.611461 22104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:56563
W20260808 21:09:12.687187 22456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:12.687431 22456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:12.687469 22456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:12.688637 22456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:12.688705 22456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:12.688751 22456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:12.690196 22456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=43317
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:12.690467 22456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:12.690708 22456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:12.695760 22464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:12.695780 22462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.695832 22456 server_base.cc:1034] running on GCE node
W20260808 21:09:12.695803 22461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:12.696182 22456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:12.696430 22456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:12.697603 22456 hybrid_clock.cc:648] HybridClock initialized: now 1786223352697592 us; error 27 us; skew 500 ppm
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:12.700489 22456 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:09:12.700943 22456 webserver.cc:460] Webserver started at http://127.4.177.67:43317/ using document root <none> and password file <none>
I20260808 21:09:12.701181 22456 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:12.701267 22456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:12.702634 22456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.703320 22471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.703495 22456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:12.703596 22456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "ab803f0628174e718313e8a18d3003a3"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6d046b38c232dd8d4be3280af4f5c746"
server_key_iv: "0b173db98e99671da9a0633f77294bc3"
server_key_version: "encryptionkey@0"
I20260808 21:09:12.704022 22456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:12.732169 22456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:12.734647 22456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:12.734826 22456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:12.735091 22456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:12.735440 22456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:12.735499 22456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.735546 22456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:12.735577 22456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.741487 22456 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46139
I20260808 21:09:12.741580 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46139 every 8 connection(s)
I20260808 21:09:12.741914 22456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:09:12 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:12.750111 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22456
I20260808 21:09:12.750926 22306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.742678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:58335 (local address 127.4.177.126:41229)
0808 21:09:12.742727 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.742729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.742944 (+ 215us) server_negotiation.cc:407] Connection header received
0808 21:09:12.743483 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.743485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.743494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.743557 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.744461 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.744953 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.745880 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.746063 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.746113 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.748211 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.748212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.748212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.748219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.749415 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.749889 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.749890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.749890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.749964 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.750198 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.750199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.750200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.750344 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.750412 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.750744 (+ 332us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.750835 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:09:12.751119 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.742645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:58335)
0808 21:09:12.742809 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.742851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.743396 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.743609 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.743612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.744184 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:12.744384 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.744389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.745031 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.745032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.745806 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.745808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.746133 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.746133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.746229 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.746764 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:12.746772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:12.748125 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:12.749486 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.749491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.749493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.749699 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.750037 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:12.750037 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:12.750037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:12.750128 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:12.750422 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:12.750428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:12.750596 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.750710 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.750879 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260808 21:09:12.751627 22595 heartbeater.cc:347] Connected to a master server at 127.4.177.126:41229
I20260808 21:09:12.751726 22595 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:12.751919 22595 heartbeater.cc:511] Master 127.4.177.126:41229 requested a full tablet report, sending...
I20260808 21:09:12.752333 22104 ts_manager.cc:194] Registered new tserver with Master: ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139)
I20260808 21:09:12.752743 22104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:58335
I20260808 21:09:12.757098 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.750568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.4.177.126:41229)
0808 21:09:12.750706 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.750708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.750808 (+ 100us) server_negotiation.cc:407] Connection header received
0808 21:09:12.750889 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.750890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.750902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.750968 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.751837 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.752402 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.753216 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.753435 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.753512 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.754680 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.754681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.754682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.754689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.755926 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.756281 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.756282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.756283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.756348 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.756555 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.756556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.756557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.756675 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.756868 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.756906 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.756990 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260808 21:09:12.757931 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:12.760494 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.758265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.4.177.126:41229)
0808 21:09:12.758323 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.758325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.758421 (+ 96us) server_negotiation.cc:407] Connection header received
0808 21:09:12.758482 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.758483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.758511 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.758570 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:12.758656 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.759091 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.759834 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.760024 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.760081 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.760262 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:12.760304 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 21:09:12.760424 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59240: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260808 21:09:12.760553 22600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59240: token expired
W20260808 21:09:12.760775 22031 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.177.126:41229)
I20260808 21:09:12.766425 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.761008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.4.177.126:41229)
0808 21:09:12.761062 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.761063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.761072 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:12.761090 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.761090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.761100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.761153 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:12.761598 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.762059 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.762714 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.762900 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.762968 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.764241 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:12.764242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:12.764242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:12.764250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:12.765477 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.765795 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.765795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.765795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.765843 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:12.766032 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:12.766032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:12.766033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:12.766127 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:12.766237 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.766302 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.766364 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:09:12.767740 22104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:12.769408 22104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:12.771817 22234 tablet_service.cc:1467] Processing CreateTablet for tablet 19027dee43b5466c8ff3fd11787c38ac (DEFAULT_TABLE table=test-table [id=dd7b18877cd94ad3a0072bb24d84265a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:12.772042 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19027dee43b5466c8ff3fd11787c38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:12.773636 22186 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Bootstrap starting.
I20260808 21:09:12.773974 22186 tablet_bootstrap.cc:654] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:12.774974 22186 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: No bootstrap required, opened a new log
I20260808 21:09:12.775027 22186 ts_tablet_manager.cc:1397] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:12.775218 22186 raft_consensus.cc:348] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.775302 22186 raft_consensus.cc:374] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:12.775352 22186 raft_consensus.cc:729] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7463732182455595be7984203f6421, State: Initialized, Role: FOLLOWER
I20260808 21:09:12.775403 22186 consensus_queue.cc:260] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.775506 22186 ts_tablet_manager.cc:1428] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:12.775648 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.771447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37807 (local address 127.0.0.1:50814)
0808 21:09:12.771790 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.771800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.771859 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.772133 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.772134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.772136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:12.772194 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.772195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.773153 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.773154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.774337 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.774339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.775374 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.775375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.775448 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.775473 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.775525 (+ 52us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.775575 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":60,"threads_started":1}
I20260808 21:09:12.775779 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.771334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46139 (local address 127.0.0.1:35436)
0808 21:09:12.771400 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.771414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.771462 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.771928 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.771929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.771931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:12.771981 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.771983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.773391 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.773392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.774434 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.774436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.775574 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.775575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.775642 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.775646 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.775676 (+ 30us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.775722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260808 21:09:12.775913 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.771578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.4.177.66:37807)
0808 21:09:12.771773 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.771777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.771844 (+ 67us) server_negotiation.cc:407] Connection header received
0808 21:09:12.771976 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.771978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.771994 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.772134 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:12.772266 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.773056 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.774428 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.775267 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.775339 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.775403 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.775767 (+ 364us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.775816 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260808 21:09:12.776437 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.771453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.4.177.67:46139)
0808 21:09:12.771738 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.771742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.771752 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:12.771790 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.771792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.771807 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.772060 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:12.772142 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.773306 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.774494 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.775460 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.775976 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.776255 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.776301 (+ 46us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.776352 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260808 21:09:12.776717 22384 tablet_service.cc:1467] Processing CreateTablet for tablet 19027dee43b5466c8ff3fd11787c38ac (DEFAULT_TABLE table=test-table [id=dd7b18877cd94ad3a0072bb24d84265a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:12.777082 22384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19027dee43b5466c8ff3fd11787c38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:12.777062 22525 tablet_service.cc:1467] Processing CreateTablet for tablet 19027dee43b5466c8ff3fd11787c38ac (DEFAULT_TABLE table=test-table [id=dd7b18877cd94ad3a0072bb24d84265a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:12.777377 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19027dee43b5466c8ff3fd11787c38ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:12.780717 22606 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Bootstrap starting.
I20260808 21:09:12.781284 22606 tablet_bootstrap.cc:654] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:12.781741 22606 log.cc:824] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:12.782131 22607 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Bootstrap starting.
I20260808 21:09:12.782524 22606 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: No bootstrap required, opened a new log
I20260808 21:09:12.782579 22606 ts_tablet_manager.cc:1397] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:12.782744 22607 tablet_bootstrap.cc:654] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:12.783291 22607 log.cc:824] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:12.784011 22607 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: No bootstrap required, opened a new log
I20260808 21:09:12.784094 22607 ts_tablet_manager.cc:1397] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:12.784399 22606 raft_consensus.cc:348] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.784533 22606 raft_consensus.cc:374] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:12.784555 22606 raft_consensus.cc:729] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31185cc22764db3959addff7a74e7d8, State: Initialized, Role: FOLLOWER
I20260808 21:09:12.784686 22606 consensus_queue.cc:260] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.784984 22606 ts_tablet_manager.cc:1428] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 21:09:12.785002 22453 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
I20260808 21:09:12.785916 22607 raft_consensus.cc:348] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.786092 22607 raft_consensus.cc:374] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:12.786144 22607 raft_consensus.cc:729] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab803f0628174e718313e8a18d3003a3, State: Initialized, Role: FOLLOWER
I20260808 21:09:12.786252 22607 consensus_queue.cc:260] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.786510 22607 ts_tablet_manager.cc:1428] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:09:12.786551 22595 heartbeater.cc:503] Master 127.4.177.126:41229 was elected leader, sending a full tablet report...
W20260808 21:09:12.850085 22454 tablet.cc:2369] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:12.859865 22612 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:12.860021 22612 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.860410 22612 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807), de7463732182455595be7984203f6421 (127.4.177.65:35181)
I20260808 21:09:12.864363 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.860759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:49117 (local address 127.4.177.66:37807)
0808 21:09:12.860894 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.860897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.860910 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:09:12.860941 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.860942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.860952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.861018 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:12.861208 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.861960 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.863420 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.863989 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.864069 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.864078 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.864247 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.864292 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260808 21:09:12.864358 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.860671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37807 (local address 127.4.177.67:49117)
0808 21:09:12.860741 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.860756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.860823 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.861026 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.861027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.861029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:12.861119 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.861121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.862058 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.862059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.863341 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.863344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.864077 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.864078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.864141 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.864164 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.864225 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.864268 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":68}
I20260808 21:09:12.864441 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.860910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:34653 (local address 127.4.177.65:35181)
0808 21:09:12.860963 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.860965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.861094 (+ 129us) server_negotiation.cc:407] Connection header received
0808 21:09:12.861146 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.861147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.861193 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.861271 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:12.861455 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.862178 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.863487 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.864103 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.864172 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.864267 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.864321 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.864363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:09:12.864490 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.860836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35181 (local address 127.4.177.67:34653)
0808 21:09:12.860999 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.861010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.861077 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.861299 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:12.861300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:12.861302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:12.861366 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.861367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.862259 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.862259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.863411 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:12.863413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.864189 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.864189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:12.864263 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.864267 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:12.864317 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:09:12.864348 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260808 21:09:12.864814 22256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19027dee43b5466c8ff3fd11787c38ac" candidate_uuid: "ab803f0628174e718313e8a18d3003a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7463732182455595be7984203f6421" is_pre_election: true
I20260808 21:09:12.864948 22406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19027dee43b5466c8ff3fd11787c38ac" candidate_uuid: "ab803f0628174e718313e8a18d3003a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31185cc22764db3959addff7a74e7d8" is_pre_election: true
I20260808 21:09:12.865005 22256 raft_consensus.cc:2454] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab803f0628174e718313e8a18d3003a3 in term 0.
I20260808 21:09:12.865128 22406 raft_consensus.cc:2454] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab803f0628174e718313e8a18d3003a3 in term 0.
I20260808 21:09:12.865268 22473 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab803f0628174e718313e8a18d3003a3, de7463732182455595be7984203f6421; no voters:
I20260808 21:09:12.865401 22612 raft_consensus.cc:2781] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:12.865459 22612 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:12.865490 22612 raft_consensus.cc:3037] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:12.866299 22612 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.866451 22612 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [CANDIDATE]: Term 1 election: Requested vote from peers d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807), de7463732182455595be7984203f6421 (127.4.177.65:35181)
I20260808 21:09:12.866595 22406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19027dee43b5466c8ff3fd11787c38ac" candidate_uuid: "ab803f0628174e718313e8a18d3003a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31185cc22764db3959addff7a74e7d8"
I20260808 21:09:12.866608 22256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19027dee43b5466c8ff3fd11787c38ac" candidate_uuid: "ab803f0628174e718313e8a18d3003a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de7463732182455595be7984203f6421"
I20260808 21:09:12.866683 22256 raft_consensus.cc:3037] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:12.866695 22406 raft_consensus.cc:3037] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:12.867573 22256 raft_consensus.cc:2454] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab803f0628174e718313e8a18d3003a3 in term 1.
I20260808 21:09:12.867664 22406 raft_consensus.cc:2454] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab803f0628174e718313e8a18d3003a3 in term 1.
I20260808 21:09:12.867733 22473 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab803f0628174e718313e8a18d3003a3, de7463732182455595be7984203f6421; no voters:
I20260808 21:09:12.867861 22612 raft_consensus.cc:2781] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:12.868028 22612 raft_consensus.cc:686] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 1 LEADER]: Becoming Leader. State: Replica: ab803f0628174e718313e8a18d3003a3, State: Running, Role: LEADER
I20260808 21:09:12.868144 22612 consensus_queue.cc:237] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:12.868932 22104 catalog_manager.cc:5495] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab803f0628174e718313e8a18d3003a3 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "ab803f0628174e718313e8a18d3003a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } health_report { overall_health: UNKNOWN } } }
I20260808 21:09:12.893038 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:12.890645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.4.177.67:46139)
0808 21:09:12.890705 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:12.890708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:12.890715 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:09:12.890757 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:12.890758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:12.890774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:12.890852 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:12.890959 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.891496 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.892256 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:12.892452 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:12.892528 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:12.892696 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:12.892793 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:12.892886 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:12.892906 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 21:09:12.892926 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:09:12.894825 22256 raft_consensus.cc:1261] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Refusing update from remote peer ab803f0628174e718313e8a18d3003a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:12.894825 22406 raft_consensus.cc:1261] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Refusing update from remote peer ab803f0628174e718313e8a18d3003a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:12.895215 22612 consensus_queue.cc:1035] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:12.895359 22615 consensus_queue.cc:1035] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:12.897325 22618 mvcc.cc:205] Tried to move back new op lower bound from 7316370853454409728 to 7316370853348556800. Current Snapshot: MvccSnapshot[applied={T|T < 7316370853454409728}]
I20260808 21:09:12.897846 22620 mvcc.cc:205] Tried to move back new op lower bound from 7316370853454409728 to 7316370853348556800. Current Snapshot: MvccSnapshot[applied={T|T < 7316370853454409728 or (T in {7316370853454409728})}]
I20260808 21:09:12.899031 22619 mvcc.cc:205] Tried to move back new op lower bound from 7316370853454409728 to 7316370853348556800. Current Snapshot: MvccSnapshot[applied={T|T < 7316370853454409728}]
I20260808 21:09:12.899729 22103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:12.899837 22103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:12.900591 22103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Sending DeleteTablet for 3 replicas of tablet 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:12.900869 22525 tablet_service.cc:1514] Processing DeleteTablet for tablet 19027dee43b5466c8ff3fd11787c38ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:12 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:35436
I20260808 21:09:12.900902 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22143
I20260808 21:09:12.901026 22384 tablet_service.cc:1514] Processing DeleteTablet for tablet 19027dee43b5466c8ff3fd11787c38ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:12 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:50814
I20260808 21:09:12.903023 22629 tablet_replica.cc:330] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: stopping tablet replica
I20260808 21:09:12.903299 22629 raft_consensus.cc:2229] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:12.903477 22629 raft_consensus.cc:2258] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:12.905095 22629 ts_tablet_manager.cc:1905] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:12.907223 22629 ts_tablet_manager.cc:1918] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:12.907249 22628 tablet_replica.cc:330] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: stopping tablet replica
I20260808 21:09:12.907349 22628 raft_consensus.cc:2229] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:12.907369 22629 log.cc:1196] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:12.907547 22628 raft_consensus.cc:2258] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:12.907676 22629 ts_tablet_manager.cc:1939] T 19027dee43b5466c8ff3fd11787c38ac P d31185cc22764db3959addff7a74e7d8: Deleting consensus metadata
I20260808 21:09:12.908145 22628 ts_tablet_manager.cc:1905] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:12.908169 22088 catalog_manager.cc:4841] TS d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): tablet 19027dee43b5466c8ff3fd11787c38ac (table test-table [id=dd7b18877cd94ad3a0072bb24d84265a]) successfully deleted
I20260808 21:09:12.909765 22628 ts_tablet_manager.cc:1918] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260808 21:09:12.909835 22088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:35181 (error 108)
I20260808 21:09:12.909870 22628 log.cc:1196] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:12.910187 22628 ts_tablet_manager.cc:1939] T 19027dee43b5466c8ff3fd11787c38ac P ab803f0628174e718313e8a18d3003a3: Deleting consensus metadata
W20260808 21:09:12.910488 22310 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:09:12.910173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:35181 (local address 127.0.0.1:44128)
0808 21:09:12.910236 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:12.910247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:12.910297 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:12.910440 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:09:12.910755 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22309
W20260808 21:09:12.910712 22088 catalog_manager.cc:4584] TS de7463732182455595be7984203f6421 (127.4.177.65:35181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19027dee43b5466c8ff3fd11787c38ac: Network error: Client connection negotiation failed: client connection to 127.4.177.65:35181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 21:09:12.911979 22088 catalog_manager.cc:4841] TS ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): tablet 19027dee43b5466c8ff3fd11787c38ac (table test-table [id=dd7b18877cd94ad3a0072bb24d84265a]) successfully deleted
I20260808 21:09:12.919544 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22456
I20260808 21:09:12.925671 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22069
I20260808 21:09:32.932542 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--webserver_interface=127.4.177.126
--webserver_port=43623
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:33.014370 22631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:33.014564 22631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:33.014586 22631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:33.015758 22631 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 21:09:33.015799 22631 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 21:09:33.015812 22631 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 21:09:33.015825 22631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:33.015880 22631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:33.015900 22631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:33.015924 22631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:33.015950 22631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:33.017468 22631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=43623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:33.017720 22631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:33.017966 22631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:33.023192 22637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.023197 22639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.023308 22631 server_base.cc:1034] running on GCE node
W20260808 21:09:33.023185 22636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.023599 22631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:33.023926 22631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:33.025087 22631 hybrid_clock.cc:648] HybridClock initialized: now 1786223373025081 us; error 36 us; skew 500 ppm
Aug 08 21:09:33 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:33.030581 22631 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:09:33.031013 22631 webserver.cc:460] Webserver started at http://127.4.177.126:43623/ using document root <none> and password file <none>
I20260808 21:09:33.031256 22631 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:33.031304 22631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:33.032526 22631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:33.033155 22646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.033339 22631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:33.033424 22631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6db95696da904e95aca51d5b49536163"
server_key_iv: "2ba87ec87847b6e7dc9fb785bd863719"
server_key_version: "encryptionkey@0"
I20260808 21:09:33.033715 22631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:33.062587 22631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:33.065547 22631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:33.065706 22631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:33.071748 22631 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:41229
I20260808 21:09:33.071785 22702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:41229 every 8 connection(s)
I20260808 21:09:33.072214 22631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:33.074190 22703 sys_catalog.cc:263] Verifying existing consensus state
I20260808 21:09:33.075238 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap starting.
I20260808 21:09:33.078487 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22631
I20260808 21:09:33.078819 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:35181
--local_ip_for_outbound_sockets=127.4.177.65
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=40523
--webserver_interface=127.4.177.65
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:09:33.086740 22703 log.cc:824] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:33.090492 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:09:33.090853 22703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Bootstrap complete.
I20260808 21:09:33.094214 22703 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:33.094534 22703 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Initialized, Role: FOLLOWER
I20260808 21:09:33.094738 22703 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:33.094812 22703 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:33.094858 22703 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:33.094897 22703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 2 FOLLOWER]: Advancing to term 3
I20260808 21:09:33.096266 22703 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:33.096381 22703 leader_election.cc:302] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb88b3d0d0146f49f8ffff4f16a36e6; no voters:
I20260808 21:09:33.096551 22703 leader_election.cc:288] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 21:09:33.096673 22709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 21:09:33.096796 22703 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:33.096900 22709 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [term 3 LEADER]: Becoming Leader. State: Replica: cdb88b3d0d0146f49f8ffff4f16a36e6, State: Running, Role: LEADER
I20260808 21:09:33.097047 22709 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } }
I20260808 21:09:33.097434 22710 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:09:33.097530 22710 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:33.097738 22719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:33.097420 22709 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb88b3d0d0146f49f8ffff4f16a36e6. Latest consensus state: current_term: 3 leader_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb88b3d0d0146f49f8ffff4f16a36e6" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 41229 } } }
I20260808 21:09:33.097898 22709 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:33.098598 22719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:33.098716 22719 catalog_manager.cc:1259] Loaded cluster ID: 94b53d1de40545159925171969f30f50
I20260808 21:09:33.103237 22719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:33.104477 22719 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:33.104671 22719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Loaded TSK: 0
I20260808 21:09:33.104709 22719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdb88b3d0d0146f49f8ffff4f16a36e6: Loaded TSK: 1
F20260808 21:09:33.104945 22719 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3579c589ad google::LogMessage::Fail() at ??:0
@ 0x7f3579c5b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3579c584de google::LogMessage::Flush() at ??:0
@ 0x7f3579c59a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f357c3c5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f357c3c2d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3581bb21cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3581bb248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3581bb2594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3581b78c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3581b82ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f357bb260d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f357bb1b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f357c97f6db start_thread at ??:0
@ 0x7f357875661f clone at ??:0
W20260808 21:09:33.183501 22707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:33.183718 22707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:33.183768 22707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:33.184914 22707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:33.184983 22707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:33.185035 22707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:33.186474 22707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:35181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=40523
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:33.186734 22707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:33.186968 22707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:33.192026 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.192010 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.192231 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.192502 22707 server_base.cc:1034] running on GCE node
I20260808 21:09:33.192715 22707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:33.193005 22707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:33.194161 22707 hybrid_clock.cc:648] HybridClock initialized: now 1786223373194137 us; error 39 us; skew 500 ppm
Aug 08 21:09:33 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:33.196697 22707 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:09:33.197121 22707 webserver.cc:460] Webserver started at http://127.4.177.65:40523/ using document root <none> and password file <none>
I20260808 21:09:33.197361 22707 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:33.197434 22707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:33.198850 22707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:33.199642 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.199826 22707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:09:33.199927 22707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "de7463732182455595be7984203f6421"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "cc02943149e664d4bd41f6a70b5d413b"
server_key_iv: "c11a62e8a113a0ce208369624534c68a"
server_key_version: "encryptionkey@0"
I20260808 21:09:33.200274 22707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:33.217514 22707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:33.219770 22707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:33.219952 22707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:33.220214 22707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:33.220669 22766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260808 21:09:33.222728 22707 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 21:09:33.222791 22707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260808 21:09:33.222836 22707 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 21:09:33.223578 22707 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 21:09:33.223637 22707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260808 21:09:33.223708 22766 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Bootstrap starting.
I20260808 21:09:33.230968 22707 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:35181
I20260808 21:09:33.231153 22882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:35181 every 8 connection(s)
I20260808 21:09:33.231431 22707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:33.235517 22766 log.cc:824] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:33.238292 22766 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 21:09:33.238647 22766 tablet_bootstrap.cc:492] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Bootstrap complete.
I20260808 21:09:33.239364 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22707
I20260808 21:09:33.239665 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:37807
--local_ip_for_outbound_sockets=127.4.177.66
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=45039
--webserver_interface=127.4.177.66
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:09:33.240036 22766 ts_tablet_manager.cc:1397] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260808 21:09:33.241472 22766 raft_consensus.cc:348] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:33.241801 22766 raft_consensus.cc:729] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7463732182455595be7984203f6421, State: Initialized, Role: FOLLOWER
I20260808 21:09:33.241976 22766 consensus_queue.cc:260] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:33.242297 22766 ts_tablet_manager.cc:1428] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260808 21:09:33.312335 22886 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:09:33.232227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:57179)
0808 21:09:33.232419 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.232454 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:33.232881 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:33.312117 (+ 79236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
W20260808 21:09:33.312582 22883 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41229 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:41229: Client connection negotiation failed: client connection to 127.4.177.126:41229: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 21:09:33.312922 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.312797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:59811)
0808 21:09:33.312853 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.312894 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 21:09:33.313329 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.313235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:60597)
0808 21:09:33.313287 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.313312 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 21:09:33.313467 22883 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 21:09:33.324812 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:33.325073 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:33.325102 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:33.326272 22888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:33.326313 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:33.326339 22888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:33.328159 22888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:37807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=45039
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:33.328400 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:33.328632 22888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:33.333644 22897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.333657 22895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.333658 22894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.333811 22888 server_base.cc:1034] running on GCE node
I20260808 21:09:33.334022 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:33.334245 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:33.335469 22888 hybrid_clock.cc:648] HybridClock initialized: now 1786223373335441 us; error 41 us; skew 500 ppm
Aug 08 21:09:33 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:33.337862 22888 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:09:33.338336 22888 webserver.cc:460] Webserver started at http://127.4.177.66:45039/ using document root <none> and password file <none>
I20260808 21:09:33.338554 22888 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:33.338625 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:33.340003 22888 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:33.340608 22904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.340771 22888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:33.340874 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "d31185cc22764db3959addff7a74e7d8"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "06a5343fb6dc98663df1a30f64a4aa49"
server_key_iv: "cedb0e6830645bb842eba08869af2cfb"
server_key_version: "encryptionkey@0"
I20260808 21:09:33.341225 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:33.349743 22888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:33.351523 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:33.351686 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:33.351922 22888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:33.352218 22888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:33.352275 22888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.352309 22888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:33.352337 22888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.358656 22888 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37807
I20260808 21:09:33.358700 23021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37807 every 8 connection(s)
I20260808 21:09:33.359010 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:33.359951 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.359601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:42973)
0808 21:09:33.359755 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.359819 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20260808 21:09:33.360363 23022 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41229 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:41229: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
I20260808 21:09:33.360749 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.360652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:38611)
0808 21:09:33.360700 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.360725 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 21:09:33.361022 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.360937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:48093)
0808 21:09:33.360979 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.361002 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 21:09:33.361128 23022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 21:09:33.365605 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 22888
I20260808 21:09:33.365885 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:46139
--local_ip_for_outbound_sockets=127.4.177.67
--tserver_master_addrs=127.4.177.126:41229
--webserver_port=43317
--webserver_interface=127.4.177.67
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:33.449494 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:33.449695 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:33.449718 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:33.450780 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:33.450814 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:33.450839 23025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:33.452338 23025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=43317
--tserver_master_addrs=127.4.177.126:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:33.452525 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:33.452695 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:33.457518 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:33.457502 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.457737 23025 server_base.cc:1034] running on GCE node
W20260808 21:09:33.457494 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:33.457988 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:33.458228 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:33.459367 23025 hybrid_clock.cc:648] HybridClock initialized: now 1786223373459360 us; error 31 us; skew 500 ppm
Aug 08 21:09:33 dist-test-slave-chsk krb5kdc[21485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:33.461887 23025 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:09:33.462363 23025 webserver.cc:460] Webserver started at http://127.4.177.67:43317/ using document root <none> and password file <none>
I20260808 21:09:33.462594 23025 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:33.462639 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:33.464010 23025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:33.464578 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.464752 23025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:33.464838 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "ab803f0628174e718313e8a18d3003a3"
format_stamp: "Formatted at 2026-08-08 21:08:51 on dist-test-slave-chsk"
server_key: "6d046b38c232dd8d4be3280af4f5c746"
server_key_iv: "0b173db98e99671da9a0633f77294bc3"
server_key_version: "encryptionkey@0"
I20260808 21:09:33.465148 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:33.481321 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:33.483636 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:33.483781 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:33.484038 23025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:33.484359 23025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:33.484414 23025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.484457 23025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:33.484485 23025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:33.490489 23025 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46139
I20260808 21:09:33.490538 23160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46139 every 8 connection(s)
I20260808 21:09:33.490872 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:33.491852 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23025
I20260808 21:09:33.492080 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.491529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:44179)
0808 21:09:33.491738 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.491875 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
W20260808 21:09:33.492596 23161 heartbeater.cc:644] Failed to heartbeat to 127.4.177.126:41229 (0 consecutive failures): Network error: Failed to ping master at 127.4.177.126:41229: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
I20260808 21:09:33.492887 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.492788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:47945)
0808 21:09:33.492837 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.492863 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:09:33.493156 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:33.493073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:36477)
0808 21:09:33.493115 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:33.493137 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260808 21:09:33.493278 23161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 21:09:34.314105 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.313883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:55445)
0808 21:09:34.314006 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:34.314041 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260808 21:09:34.361621 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.361419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:46523)
0808 21:09:34.361544 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:34.361576 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260808 21:09:34.493918 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.493597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:54449)
0808 21:09:34.493792 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:34.493824 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260808 21:09:34.964146 23168 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:34.964311 23168 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:34.964633 23168 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
Aug 08 21:09:34 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.67@KRBTEST.COM
Aug 08 21:09:34 dist-test-slave-chsk krb5kdc[21485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223373, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.66@KRBTEST.COM
I20260808 21:09:34.975457 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.964891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:39493 (local address 127.4.177.67:46139)
0808 21:09:34.965096 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:34.965124 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:34.965179 (+ 55us) server_negotiation.cc:407] Connection header received
0808 21:09:34.965752 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:34.965755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:34.965809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:34.965950 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:34.966772 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.968153 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969111 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.969511 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969636 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:34.971999 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:34.972002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:34.972003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:34.972024 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:34.973915 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974302 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:34.974305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:34.974305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:34.974394 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974645 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:34.974646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:34.974646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:34.974860 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:34.974943 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:34.975225 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 21:09:34.975345 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260808 21:09:34.975463 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.964786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46139 (local address 127.4.177.65:39493)
0808 21:09:34.964892 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:34.964902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:34.964956 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:34.966002 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:34.966004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:34.966461 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:34.966641 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:34.966647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.968242 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.968242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:34.969014 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:34.969017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.969608 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:34.969717 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:34.970292 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:34.970302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:34.971897 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:34.973993 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:34.974001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:34.974201 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:34.974463 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:34.974464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:34.974565 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:34.974936 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:34.974942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:34.975053 (+ 111us) client_negotiation.cc:769] Sending connection context
0808 21:09:34.975166 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 21:09:34.975330 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":28,"thread_start_us":70,"threads_started":1}
I20260808 21:09:34.975540 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.964960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37807 (local address 127.4.177.65:52141)
0808 21:09:34.965073 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:34.965085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:34.965166 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:34.966069 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:34.966070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:34.966462 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:34.966642 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:34.966647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.968279 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.968280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:34.969007 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:34.969009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.969793 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:34.969835 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:34.970432 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:34.970436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:34.972255 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:34.974234 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:34.974236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:34.974393 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:34.974641 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:34.974642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:34.974704 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:34.975055 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:34.975055 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:34.975096 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 21:09:34.975201 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 21:09:34.975404 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":123,"thread_start_us":68,"threads_started":1}
I20260808 21:09:34.976015 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:34.965055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:52141 (local address 127.4.177.66:37807)
0808 21:09:34.965214 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:34.965221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:34.965239 (+ 18us) server_negotiation.cc:407] Connection header received
0808 21:09:34.965751 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:34.965755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:34.965810 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:34.965950 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:34.966788 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.968154 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969105 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:34.969515 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:34.969632 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:34.972359 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:34.972362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:34.972362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:34.972371 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:34.974153 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974481 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:34.974484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:34.974484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:34.974575 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:34.974771 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:34.974772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:34.974772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:34.974986 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:34.975071 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:34.975259 (+ 188us) server_negotiation.cc:299] Negotiation successful
0808 21:09:34.975403 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20260808 21:09:34.976259 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
W20260808 21:09:34.976636 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:34.976691 22761 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7463732182455595be7984203f6421; no voters: ab803f0628174e718313e8a18d3003a3, d31185cc22764db3959addff7a74e7d8
I20260808 21:09:34.976809 23168 raft_consensus.cc:2726] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:09:35.314716 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:35.314554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:39955)
0808 21:09:35.314627 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:35.314673 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 21:09:35.362280 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:35.362067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:47617)
0808 21:09:35.362191 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:35.362238 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260808 21:09:35.494585 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:35.494370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:45233)
0808 21:09:35.494511 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:35.494546 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260808 21:09:36.315478 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:36.315261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:54707)
0808 21:09:36.315386 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:36.315428 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260808 21:09:36.362977 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:36.362749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:43029)
0808 21:09:36.362891 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:36.362932 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260808 21:09:36.495364 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:36.495099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:50883)
0808 21:09:36.495281 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:36.495322 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":127,"threads_started":1}
I20260808 21:09:37.181795 23181 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:37.181900 23181 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:37.182118 23181 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
W20260808 21:09:37.182590 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
W20260808 21:09:37.182680 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:37.182719 22761 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7463732182455595be7984203f6421; no voters: ab803f0628174e718313e8a18d3003a3, d31185cc22764db3959addff7a74e7d8
I20260808 21:09:37.182812 23181 raft_consensus.cc:2726] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:09:37.316212 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:37.315958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:35253)
0808 21:09:37.316124 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:37.316163 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260808 21:09:37.363675 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:37.363457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:37843)
0808 21:09:37.363581 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:37.363620 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260808 21:09:37.495990 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:37.495770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:34405)
0808 21:09:37.495903 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:37.495934 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260808 21:09:38.316884 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:38.316671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:58687)
0808 21:09:38.316793 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:38.316830 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260808 21:09:38.364358 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:38.364148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:55513)
0808 21:09:38.364282 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:38.364315 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260808 21:09:38.496675 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:38.496428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:33215)
0808 21:09:38.496597 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:38.496635 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260808 21:09:39.317612 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:39.317346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:33141)
0808 21:09:39.317525 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:39.317564 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260808 21:09:39.365068 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:39.364838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:35783)
0808 21:09:39.364981 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:39.365025 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260808 21:09:39.497449 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:39.497216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:37647)
0808 21:09:39.497365 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:39.497403 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260808 21:09:39.791464 23193 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:39.791563 23193 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:39.791715 23193 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
W20260808 21:09:39.792138 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
W20260808 21:09:39.792207 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:39.792246 22761 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7463732182455595be7984203f6421; no voters: ab803f0628174e718313e8a18d3003a3, d31185cc22764db3959addff7a74e7d8
I20260808 21:09:39.792327 23193 raft_consensus.cc:2726] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:09:40.318235 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:40.318016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:44681)
0808 21:09:40.318138 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:40.318174 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260808 21:09:40.365824 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:40.365585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:45281)
0808 21:09:40.365731 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:40.365779 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260808 21:09:40.498150 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:40.497910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:50611)
0808 21:09:40.498071 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:40.498108 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260808 21:09:41.318892 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:41.318635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:34917)
0808 21:09:41.318793 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:41.318836 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260808 21:09:41.366530 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:41.366271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:34895)
0808 21:09:41.366449 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:41.366488 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260808 21:09:41.498819 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:41.498606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:38331)
0808 21:09:41.498734 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:41.498774 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260808 21:09:42.319780 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:42.319464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:49453)
0808 21:09:42.319616 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:42.319670 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260808 21:09:42.367368 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:42.367126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:54939)
0808 21:09:42.367293 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:42.367327 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
I20260808 21:09:42.499505 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:42.499279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:44875)
0808 21:09:42.499427 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:42.499464 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260808 21:09:42.548331 23203 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:42.548422 23203 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:42.548602 23203 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
W20260808 21:09:42.548962 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
W20260808 21:09:42.549052 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:42.549084 22761 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7463732182455595be7984203f6421; no voters: ab803f0628174e718313e8a18d3003a3, d31185cc22764db3959addff7a74e7d8
I20260808 21:09:42.549170 23203 raft_consensus.cc:2726] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:09:43.320596 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:43.320349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:35017)
0808 21:09:43.320502 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:43.320547 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260808 21:09:43.367940 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:43.367728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:38435)
0808 21:09:43.367835 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:43.367897 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260808 21:09:43.500180 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:43.499975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:55469)
0808 21:09:43.500104 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:43.500140 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260808 21:09:44.321341 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:44.321093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:51989)
0808 21:09:44.321248 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:44.321290 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260808 21:09:44.368530 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:44.368343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:57725)
0808 21:09:44.368453 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:44.368486 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260808 21:09:44.500886 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:44.500658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:44299)
0808 21:09:44.500805 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:44.500843 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260808 21:09:45.322194 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:45.321947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:33491)
0808 21:09:45.322093 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:45.322130 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260808 21:09:45.369356 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:45.369112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:60511)
0808 21:09:45.369265 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:45.369306 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260808 21:09:45.501693 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:45.501339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:46575)
0808 21:09:45.501609 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:45.501639 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":214,"threads_started":1}
I20260808 21:09:46.322885 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:46.322639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:35245)
0808 21:09:46.322790 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:46.322833 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260808 21:09:46.370021 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:46.369789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:51247)
0808 21:09:46.369937 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:46.369977 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260808 21:09:46.502345 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:46.502116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:34961)
0808 21:09:46.502248 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:46.502284 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260808 21:09:47.323664 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:47.323409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:38417)
0808 21:09:47.323569 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:47.323612 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260808 21:09:47.370782 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:47.370548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:54935)
0808 21:09:47.370691 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:47.370732 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260808 21:09:47.503189 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:47.502887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:38235)
0808 21:09:47.503039 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:47.503083 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260808 21:09:47.870191 23221 raft_consensus.cc:482] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:47.870314 23221 raft_consensus.cc:504] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab803f0628174e718313e8a18d3003a3" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46139 } } peers { permanent_uuid: "d31185cc22764db3959addff7a74e7d8" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37807 } } peers { permanent_uuid: "de7463732182455595be7984203f6421" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 35181 } }
I20260808 21:09:47.870503 23221 leader_election.cc:288] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139), d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807)
W20260808 21:09:47.870919 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ab803f0628174e718313e8a18d3003a3 (127.4.177.67:46139): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
W20260808 21:09:47.871001 22761 leader_election.cc:341] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d31185cc22764db3959addff7a74e7d8 (127.4.177.66:37807): Not found: Tablet not found: 19027dee43b5466c8ff3fd11787c38ac
I20260808 21:09:47.871047 22761 leader_election.cc:302] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de7463732182455595be7984203f6421; no voters: ab803f0628174e718313e8a18d3003a3, d31185cc22764db3959addff7a74e7d8
I20260808 21:09:47.871126 23221 raft_consensus.cc:2726] T 19027dee43b5466c8ff3fd11787c38ac P de7463732182455595be7984203f6421 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 21:09:48.324544 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:48.324206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.65:40057)
0808 21:09:48.324400 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:48.324479 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260808 21:09:48.371548 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:48.371295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.66:37581)
0808 21:09:48.371438 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:48.371477 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260808 21:09:48.498958 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22707
I20260808 21:09:48.503885 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:48.503682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:41229 (local address 127.4.177.67:49719)
0808 21:09:48.503811 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:48.503843 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.126:41229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260808 21:09:48.505339 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 22888
I20260808 21:09:48.512064 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23025
2026-08-08T21:09:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58480 ms)
[----------] 30 tests from SecurityITest (232957 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:48 dist-test-slave-chsk krb5kdc[23227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:48 dist-test-slave-chsk krb5kdc[23227](info): set up 2 sockets
Aug 08 21:09:48 dist-test-slave-chsk krb5kdc[23227](info): commencing operation
krb5kdc: starting...
W20260808 21:09:49.591965 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:09:49 dist-test-slave-chsk krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:09:49.716137 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40667
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:49.802565 23240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:49.802807 23240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:49.802845 23240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:49.804045 23240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:49.804111 23240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:49.804136 23240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:49.804157 23240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:49.804188 23240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:49.805648 23240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:49.805927 23240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:49.806156 23240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:49.811321 23246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:49.811326 23245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:49.811331 23248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:49.811709 23240 server_base.cc:1034] running on GCE node
I20260808 21:09:49.811955 23240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:49.812196 23240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:49.813339 23240 hybrid_clock.cc:648] HybridClock initialized: now 1786223389813325 us; error 30 us; skew 500 ppm
Aug 08 21:09:49 dist-test-slave-chsk krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:49.815871 23240 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:09:49.816339 23240 webserver.cc:460] Webserver started at http://127.4.177.126:42767/ using document root <none> and password file <none>
I20260808 21:09:49.816527 23240 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:49.816568 23240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:49.816659 23240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:49.817662 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "a614dd5716364c13987931cdffd6065f"
format_stamp: "Formatted at 2026-08-08 21:09:49 on dist-test-slave-chsk"
server_key: "847db7f121c60d5bd35fd42fdfad5731"
server_key_iv: "d66df0194c2224acee9cb03446fb68b7"
server_key_version: "encryptionkey@0"
I20260808 21:09:49.817979 23240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "a614dd5716364c13987931cdffd6065f"
format_stamp: "Formatted at 2026-08-08 21:09:49 on dist-test-slave-chsk"
server_key: "847db7f121c60d5bd35fd42fdfad5731"
server_key_iv: "d66df0194c2224acee9cb03446fb68b7"
server_key_version: "encryptionkey@0"
I20260808 21:09:49.819351 23240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:49.820070 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:49.820261 23240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:49.820350 23240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "a614dd5716364c13987931cdffd6065f"
format_stamp: "Formatted at 2026-08-08 21:09:49 on dist-test-slave-chsk"
server_key: "847db7f121c60d5bd35fd42fdfad5731"
server_key_iv: "d66df0194c2224acee9cb03446fb68b7"
server_key_version: "encryptionkey@0"
I20260808 21:09:49.820401 23240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:49.829381 23240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:49.831933 23240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:49.832051 23240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:49.836028 23240 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:40667
I20260808 21:09:49.836077 23310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:40667 every 8 connection(s)
I20260808 21:09:49.836432 23240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:49.837359 23311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:49.839840 23311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: Bootstrap starting.
I20260808 21:09:49.840456 23311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:49.840751 23311 log.cc:824] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:49.841512 23311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: No bootstrap required, opened a new log
I20260808 21:09:49.841660 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23240
I20260808 21:09:49.841780 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:49.841979 4805 external_mini_cluster.cc:1428] Setting key ae579ddb0bec2771f975fe05f5877d1b
I20260808 21:09:49.843742 23311 raft_consensus.cc:348] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } }
I20260808 21:09:49.843917 23311 raft_consensus.cc:374] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:49.844023 23311 raft_consensus.cc:729] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a614dd5716364c13987931cdffd6065f, State: Initialized, Role: FOLLOWER
I20260808 21:09:49.844204 23311 consensus_queue.cc:260] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } }
I20260808 21:09:49.844313 23311 raft_consensus.cc:388] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:49.844353 23311 raft_consensus.cc:482] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:49.844411 23311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:49.845036 23311 raft_consensus.cc:504] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } }
I20260808 21:09:49.845163 23311 leader_election.cc:302] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a614dd5716364c13987931cdffd6065f; no voters:
I20260808 21:09:49.845341 23311 leader_election.cc:288] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:49.845402 23316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:49.845546 23316 raft_consensus.cc:686] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [term 1 LEADER]: Becoming Leader. State: Replica: a614dd5716364c13987931cdffd6065f, State: Running, Role: LEADER
I20260808 21:09:49.845608 23311 sys_catalog.cc:564] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:49.845675 23316 consensus_queue.cc:237] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } }
I20260808 21:09:49.846002 23316 sys_catalog.cc:455] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a614dd5716364c13987931cdffd6065f. Latest consensus state: current_term: 1 leader_uuid: "a614dd5716364c13987931cdffd6065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } } }
I20260808 21:09:49.846120 23316 sys_catalog.cc:458] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:49.846014 23317 sys_catalog.cc:455] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a614dd5716364c13987931cdffd6065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a614dd5716364c13987931cdffd6065f" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 40667 } } }
I20260808 21:09:49.846261 23317 sys_catalog.cc:458] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:49.847162 23323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:49.847782 23323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 21:09:49 dist-test-slave-chsk krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:49.851083 23323 catalog_manager.cc:1347] Generated new cluster ID: 72ee9338f48b406dacd589005c49a42a
I20260808 21:09:49.851135 23323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:49.854017 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:49.842490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.4.177.126:40667)
0808 21:09:49.842694 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:49.842702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:49.842804 (+ 102us) server_negotiation.cc:407] Connection header received
0808 21:09:49.843720 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:49.843726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:49.843772 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:49.843883 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:49.844687 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:49.846286 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:49.847432 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:49.847832 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:49.847916 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:49.850579 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:49.850582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:49.850583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:49.850596 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:49.852410 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:49.852841 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:49.852844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:49.852845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:49.852964 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:49.853187 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:49.853188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:49.853188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:49.853449 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:49.853632 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:49.853697 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 21:09:49.853842 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260808 21:09:49.862705 23323 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:49.863300 23323 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:49.865164 23323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: Generated new TSK 0
I20260808 21:09:49.865363 23323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:09:49.898864 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:50.001137 23338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:50.001343 23338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:50.001367 23338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:50.002519 23338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:50.002561 23338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:50.002588 23338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:50.004089 23338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:50.004387 23338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:50.004624 23338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:50.009505 23343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.009614 23346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.009548 23344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.009899 23338 server_base.cc:1034] running on GCE node
I20260808 21:09:50.010110 23338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:50.010354 23338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:50.011518 23338 hybrid_clock.cc:648] HybridClock initialized: now 1786223390011494 us; error 41 us; skew 500 ppm
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:50.013968 23338 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:09:50.014488 23338 webserver.cc:460] Webserver started at http://127.4.177.65:42285/ using document root <none> and password file <none>
I20260808 21:09:50.014719 23338 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:50.014782 23338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:50.014918 23338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:50.016047 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "8526fd24dcd0468c8106f01fb5cb81e4"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d80bdb2c14f779c33295600d6d9887d2"
server_key_iv: "f1727d4ec1d2662d3bf3bf4fd9436ac3"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.016420 23338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "8526fd24dcd0468c8106f01fb5cb81e4"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d80bdb2c14f779c33295600d6d9887d2"
server_key_iv: "f1727d4ec1d2662d3bf3bf4fd9436ac3"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.017756 23338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:09:50.018476 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.018663 23338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:50.018759 23338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "8526fd24dcd0468c8106f01fb5cb81e4"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d80bdb2c14f779c33295600d6d9887d2"
server_key_iv: "f1727d4ec1d2662d3bf3bf4fd9436ac3"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.018841 23338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:50.028883 23338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:50.030699 23338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:50.030861 23338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:50.031090 23338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:50.031427 23338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:50.031481 23338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.031523 23338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:50.031550 23338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.037611 23338 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:36481
I20260808 21:09:50.037655 23473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:36481 every 8 connection(s)
I20260808 21:09:50.037955 23338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:50.044875 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23338
I20260808 21:09:50.044984 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:50.045248 4805 external_mini_cluster.cc:1428] Setting key f221f1063edd53e918bf4a2747b2adf8
I20260808 21:09:50.047961 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.038665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:51997 (local address 127.4.177.126:40667)
0808 21:09:50.038719 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.038721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.038914 (+ 193us) server_negotiation.cc:407] Connection header received
0808 21:09:50.039521 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.039522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.039540 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.039599 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:50.040577 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.041096 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.042112 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.042337 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.042418 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.044567 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:50.044568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:50.044569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:50.044576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:50.046410 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.046793 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.046794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.046795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.046864 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.047207 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.047208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.047209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.047337 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:50.047395 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.047729 (+ 334us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.047821 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:09:50.048166 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.038620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40667 (local address 127.4.177.65:51997)
0808 21:09:50.038783 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.038825 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.039411 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.039675 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.039678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.040272 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:50.040466 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.040470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.041204 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.041206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.042005 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.042007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.042439 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.042440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.042562 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.043044 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:50.043051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:50.044471 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:50.046488 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.046494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.046496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.046715 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.046950 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.046951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.046952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.047111 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.047405 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:50.047412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:50.047585 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.047694 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.047880 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260808 21:09:50.048831 23474 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40667
I20260808 21:09:50.048946 23474 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:50.049165 23474 heartbeater.cc:511] Master 127.4.177.126:40667 requested a full tablet report, sending...
I20260808 21:09:50.049703 23275 ts_manager.cc:194] Registered new tserver with Master: 8526fd24dcd0468c8106f01fb5cb81e4 (127.4.177.65:36481)
I20260808 21:09:50.050324 23275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:51997
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:09:50.073416 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:50.155915 23479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:50.156145 23479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:50.156185 23479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:50.157320 23479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:50.157382 23479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:50.157426 23479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:50.158874 23479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:50.159129 23479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:50.159385 23479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:50.164729 23485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.164732 23487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.164851 23479 server_base.cc:1034] running on GCE node
W20260808 21:09:50.164736 23484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.165235 23479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:50.165510 23479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:50.166683 23479 hybrid_clock.cc:648] HybridClock initialized: now 1786223390166668 us; error 26 us; skew 500 ppm
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:50.169410 23479 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:09:50.169817 23479 webserver.cc:460] Webserver started at http://127.4.177.66:44101/ using document root <none> and password file <none>
I20260808 21:09:50.170009 23479 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:50.170051 23479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:50.170162 23479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:50.171237 23479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "6d7e3464065d43e2b30b316b932acf20"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d4aa17f29c084e1c11758b77f85ef7d3"
server_key_iv: "bea08afdc13602c90dc018c788ad71ab"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.171552 23479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "6d7e3464065d43e2b30b316b932acf20"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d4aa17f29c084e1c11758b77f85ef7d3"
server_key_iv: "bea08afdc13602c90dc018c788ad71ab"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.172960 23479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:50.173686 23494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.173916 23479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:50.174017 23479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "6d7e3464065d43e2b30b316b932acf20"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "d4aa17f29c084e1c11758b77f85ef7d3"
server_key_iv: "bea08afdc13602c90dc018c788ad71ab"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.174096 23479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:50.195739 23479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:50.198184 23479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:50.198333 23479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:50.198603 23479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:50.198994 23479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:50.199048 23479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.199092 23479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:50.199129 23479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.204751 23479 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:38673
I20260808 21:09:50.204802 23617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:38673 every 8 connection(s)
I20260808 21:09:50.205137 23479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:50.209288 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23479
I20260808 21:09:50.209416 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:50.209620 4805 external_mini_cluster.cc:1428] Setting key fe803dd8b62264363b5fa15dd274ddf9
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:50.215193 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.205867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:45981 (local address 127.4.177.126:40667)
0808 21:09:50.205936 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.205939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.206158 (+ 219us) server_negotiation.cc:407] Connection header received
0808 21:09:50.206695 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.206696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.206709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.206766 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:50.207834 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.208355 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.209362 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.209625 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.209706 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.212297 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:50.212298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:50.212299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:50.212306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:50.213763 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.214174 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.214175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.214176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.214239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.214492 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.214493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.214494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.214621 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:50.214695 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.214986 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.215076 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:09:50.215407 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.205847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40667 (local address 127.4.177.66:45981)
0808 21:09:50.206028 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.206070 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.206586 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.206837 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.206841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.207453 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:50.207728 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.207734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.208459 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.208460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.209271 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.209274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.209711 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.209712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.209879 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.210548 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:50.210563 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:50.212198 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:50.213857 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.213864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.213867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.214097 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.214313 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.214313 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.214314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.214427 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.214697 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:50.214724 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:50.214857 (+ 133us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.214965 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.215160 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260808 21:09:50.215873 23618 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40667
I20260808 21:09:50.215957 23618 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:50.216156 23618 heartbeater.cc:511] Master 127.4.177.126:40667 requested a full tablet report, sending...
I20260808 21:09:50.216589 23275 ts_manager.cc:194] Registered new tserver with Master: 6d7e3464065d43e2b30b316b932acf20 (127.4.177.66:38673)
I20260808 21:09:50.217013 23275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:45981
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:09:50.236881 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:50.320686 23623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:50.320883 23623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:50.320907 23623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:50.322157 23623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:50.322234 23623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:50.322294 23623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:50.323973 23623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:50.324208 23623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:50.324438 23623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:50.330019 23631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.330020 23629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.330077 23628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.330277 23623 server_base.cc:1034] running on GCE node
I20260808 21:09:50.330474 23623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:50.330672 23623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:50.331811 23623 hybrid_clock.cc:648] HybridClock initialized: now 1786223390331802 us; error 30 us; skew 500 ppm
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:50.334389 23623 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:09:50.334852 23623 webserver.cc:460] Webserver started at http://127.4.177.67:35091/ using document root <none> and password file <none>
I20260808 21:09:50.335034 23623 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:50.335078 23623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:50.335237 23623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:50.336299 23623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "ff62e938ad664f8f8e6d54cd3ae2425a"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "3aa306dccdc2a281bf7ed13490ea638e"
server_key_iv: "7e836eb3656c73224efa1072e1d589b8"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.336668 23623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "ff62e938ad664f8f8e6d54cd3ae2425a"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "3aa306dccdc2a281bf7ed13490ea638e"
server_key_iv: "7e836eb3656c73224efa1072e1d589b8"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.338019 23623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:50.338742 23638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.338917 23623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:50.339025 23623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "ff62e938ad664f8f8e6d54cd3ae2425a"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "3aa306dccdc2a281bf7ed13490ea638e"
server_key_iv: "7e836eb3656c73224efa1072e1d589b8"
server_key_version: "encryptionkey@0"
I20260808 21:09:50.339103 23623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:50.363456 23623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:50.365540 23623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:50.365715 23623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:50.365972 23623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:50.366297 23623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:50.366353 23623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.366418 23623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:50.366447 23623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:50.372537 23623 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:35729
I20260808 21:09:50.372581 23758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:35729 every 8 connection(s)
I20260808 21:09:50.372962 23623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
Aug 08 21:09:50 dist-test-slave-chsk krb5kdc[23227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:50.382083 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.373725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:33297 (local address 127.4.177.126:40667)
0808 21:09:50.373785 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.373787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.373982 (+ 195us) server_negotiation.cc:407] Connection header received
0808 21:09:50.374532 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.374533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.374544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.374616 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:50.375620 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.376131 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.377162 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.377369 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.377424 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.379563 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:50.379564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:50.379564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:50.379570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:50.380955 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.381306 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.381306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.381307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.381356 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.381563 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.381563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.381564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.381662 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:50.381710 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.381933 (+ 223us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.381990 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:09:50.382246 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.373691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:40667 (local address 127.4.177.67:33297)
0808 21:09:50.373849 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.373892 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.374418 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.374686 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.374690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.375326 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:50.375525 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.375529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.376209 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.376210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.377078 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.377081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.377441 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.377442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.377537 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.378066 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:50.378075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:50.379480 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:50.381024 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.381029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.381030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.381242 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.381414 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:50.381414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:50.381414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:50.381508 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:50.381715 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:50.381719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:50.381827 (+ 108us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.381906 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.382036 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260808 21:09:50.382741 23759 heartbeater.cc:347] Connected to a master server at 127.4.177.126:40667
I20260808 21:09:50.382855 23759 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:50.383044 23759 heartbeater.cc:511] Master 127.4.177.126:40667 requested a full tablet report, sending...
I20260808 21:09:50.383144 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23623
I20260808 21:09:50.383291 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:50.383503 4805 external_mini_cluster.cc:1428] Setting key 10892cf6e7e888ab9554fb1ebac049a4
I20260808 21:09:50.383508 23275 ts_manager.cc:194] Registered new tserver with Master: ff62e938ad664f8f8e6d54cd3ae2425a (127.4.177.67:35729)
I20260808 21:09:50.383992 23275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:33297
I20260808 21:09:50.384276 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:50.391119 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.385542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.4.177.126:40667)
0808 21:09:50.385582 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.385584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.385724 (+ 140us) server_negotiation.cc:407] Connection header received
0808 21:09:50.385759 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.385759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.385768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.385808 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:50.386308 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.386747 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.387511 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.387694 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.387759 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.388785 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:50.388786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:50.388786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:50.388792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:50.390144 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.390461 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.390462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.390462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.390510 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:50.390673 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:50.390673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:50.390674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:50.390810 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:50.390887 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.390970 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.391029 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:09:50.392705 23274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:50.394230 23274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:50.400842 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36481 (local address 127.0.0.1:54038)
0808 21:09:50.396944 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.396972 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.397012 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.397332 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.397335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.397338 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:50.397413 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.397416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.398716 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.398717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.399858 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.399860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.400534 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.400535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.400624 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.400638 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.400706 (+ 68us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.400766 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260808 21:09:50.401172 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.4.177.65:36481)
0808 21:09:50.397065 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.397074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.397084 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:50.397128 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.397130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.397147 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.397234 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:50.397483 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.398604 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.399935 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.400440 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.400531 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.400592 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.401053 (+ 461us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.401096 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260808 21:09:50.401695 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:35729 (local address 127.0.0.1:54656)
0808 21:09:50.396742 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.396787 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.396852 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.397615 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.397616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.397619 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:50.397656 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.397658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.398475 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.398476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.399843 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.399847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.401451 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.401452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.401520 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.401525 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.401576 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.401621 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260808 21:09:50.401940 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.4.177.67:35729)
0808 21:09:50.397155 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.397158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.397165 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:09:50.397196 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.397217 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.397236 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.397621 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:50.397717 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.398376 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.400630 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.401377 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.401449 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.401789 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.401814 (+ 25us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.401855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":100,"threads_started":1}
I20260808 21:09:50.401845 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:38673 (local address 127.0.0.1:38608)
0808 21:09:50.397082 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.397091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.397134 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.397353 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.397354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.397357 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:50.397432 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.397434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.399943 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.399943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.400631 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.400633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.401633 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.401634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.401691 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.401694 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.401743 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.401785 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":96,"threads_started":1}
I20260808 21:09:50.401888 23404 tablet_service.cc:1467] Processing CreateTablet for tablet 3d65af3d683e4c89a066ab440e9a296d (DEFAULT_TABLE table=test-table [id=7d8225938f5745eeae934ea389b6775b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:50.402091 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.396849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.4.177.66:38673)
0808 21:09:50.397065 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.397079 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.397200 (+ 121us) server_negotiation.cc:407] Connection header received
0808 21:09:50.397257 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.397259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.397274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.397346 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:50.397518 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.398604 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.400731 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.401474 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.401543 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.401600 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.401907 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.401954 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
I20260808 21:09:50.402316 23404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d65af3d683e4c89a066ab440e9a296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:50.402653 23546 tablet_service.cc:1467] Processing CreateTablet for tablet 3d65af3d683e4c89a066ab440e9a296d (DEFAULT_TABLE table=test-table [id=7d8225938f5745eeae934ea389b6775b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:50.402956 23546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d65af3d683e4c89a066ab440e9a296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:50.403782 23690 tablet_service.cc:1467] Processing CreateTablet for tablet 3d65af3d683e4c89a066ab440e9a296d (DEFAULT_TABLE table=test-table [id=7d8225938f5745eeae934ea389b6775b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:50.404109 23690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d65af3d683e4c89a066ab440e9a296d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:50.405520 23777 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Bootstrap starting.
I20260808 21:09:50.406183 23777 tablet_bootstrap.cc:654] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:50.406311 23778 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Bootstrap starting.
I20260808 21:09:50.406883 23777 log.cc:824] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:50.407032 23778 tablet_bootstrap.cc:654] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:50.407337 23780 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Bootstrap starting.
I20260808 21:09:50.407635 23778 log.cc:824] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:50.407748 23777 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: No bootstrap required, opened a new log
I20260808 21:09:50.407857 23777 ts_tablet_manager.cc:1397] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:50.408128 23780 tablet_bootstrap.cc:654] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:50.408597 23778 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: No bootstrap required, opened a new log
I20260808 21:09:50.408671 23778 ts_tablet_manager.cc:1397] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:09:50.408629 23780 log.cc:824] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:50.409308 23780 tablet_bootstrap.cc:492] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: No bootstrap required, opened a new log
I20260808 21:09:50.409351 23780 ts_tablet_manager.cc:1397] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:50.409941 23777 raft_consensus.cc:348] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.410131 23777 raft_consensus.cc:374] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:50.410169 23777 raft_consensus.cc:729] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8526fd24dcd0468c8106f01fb5cb81e4, State: Initialized, Role: FOLLOWER
I20260808 21:09:50.410290 23777 consensus_queue.cc:260] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.410584 23777 ts_tablet_manager.cc:1428] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:09:50.410640 23474 heartbeater.cc:503] Master 127.4.177.126:40667 was elected leader, sending a full tablet report...
I20260808 21:09:50.410679 23778 raft_consensus.cc:348] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.410826 23778 raft_consensus.cc:374] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:50.410849 23778 raft_consensus.cc:729] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7e3464065d43e2b30b316b932acf20, State: Initialized, Role: FOLLOWER
I20260808 21:09:50.410846 23780 raft_consensus.cc:348] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.410957 23778 consensus_queue.cc:260] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.411031 23780 raft_consensus.cc:374] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:50.411120 23780 raft_consensus.cc:729] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff62e938ad664f8f8e6d54cd3ae2425a, State: Initialized, Role: FOLLOWER
I20260808 21:09:50.411273 23618 heartbeater.cc:503] Master 127.4.177.126:40667 was elected leader, sending a full tablet report...
I20260808 21:09:50.411269 23778 ts_tablet_manager.cc:1428] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260808 21:09:50.411371 23780 consensus_queue.cc:260] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.411643 23780 ts_tablet_manager.cc:1428] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:50.411758 23759 heartbeater.cc:503] Master 127.4.177.126:40667 was elected leader, sending a full tablet report...
W20260808 21:09:50.455789 23619 tablet.cc:2369] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:09:50.538646 23475 tablet.cc:2369] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:09:50.623595 23760 tablet.cc:2369] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:50.769205 23784 raft_consensus.cc:482] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:50.769330 23784 raft_consensus.cc:504] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.769750 23784 leader_election.cc:288] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff62e938ad664f8f8e6d54cd3ae2425a (127.4.177.67:35729), 8526fd24dcd0468c8106f01fb5cb81e4 (127.4.177.65:36481)
I20260808 21:09:50.773761 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.770105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:52591 (local address 127.4.177.65:36481)
0808 21:09:50.770151 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.770153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.770161 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:50.770198 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.770199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.770209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.770276 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:50.770535 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.771320 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.772848 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.773403 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.773480 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.773490 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.773655 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.773691 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260808 21:09:50.773761 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.770064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:35893 (local address 127.4.177.67:35729)
0808 21:09:50.770122 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.770124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.770137 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:09:50.770234 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.770235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.770244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.770296 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:50.770535 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.771329 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.772848 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.773403 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.773481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.773490 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.773655 (+ 165us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.773691 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:09:50.773860 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.769941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36481 (local address 127.4.177.66:52591)
0808 21:09:50.770081 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.770090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.770146 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.770309 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.770310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.770313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:50.770435 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.770437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.771410 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.771411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.772753 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.772757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.773488 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.773490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.773568 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.773574 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.773633 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.773688 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":96,"thread_start_us":63,"threads_started":1}
I20260808 21:09:50.773859 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.769941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:35729 (local address 127.4.177.66:35893)
0808 21:09:50.770074 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.770087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:50.770146 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:50.770309 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:50.770310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:50.770313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:50.770439 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.770440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.771408 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.771409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.772763 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:50.772765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.773490 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.773491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:50.773568 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.773576 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 21:09:50.773633 (+ 57us) client_negotiation.cc:240] Negotiation successful
0808 21:09:50.773687 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":99,"thread_start_us":64,"threads_started":1}
I20260808 21:09:50.774214 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d65af3d683e4c89a066ab440e9a296d" candidate_uuid: "6d7e3464065d43e2b30b316b932acf20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" is_pre_election: true
I20260808 21:09:50.774369 23425 raft_consensus.cc:2454] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e3464065d43e2b30b316b932acf20 in term 0.
I20260808 21:09:50.774459 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d65af3d683e4c89a066ab440e9a296d" candidate_uuid: "6d7e3464065d43e2b30b316b932acf20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" is_pre_election: true
I20260808 21:09:50.774618 23710 raft_consensus.cc:2454] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7e3464065d43e2b30b316b932acf20 in term 0.
I20260808 21:09:50.774626 23497 leader_election.cc:302] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d7e3464065d43e2b30b316b932acf20, 8526fd24dcd0468c8106f01fb5cb81e4; no voters:
I20260808 21:09:50.774748 23784 raft_consensus.cc:2781] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:50.774808 23784 raft_consensus.cc:482] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:50.774832 23784 raft_consensus.cc:3037] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:50.775568 23784 raft_consensus.cc:504] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.775720 23784 leader_election.cc:288] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [CANDIDATE]: Term 1 election: Requested vote from peers ff62e938ad664f8f8e6d54cd3ae2425a (127.4.177.67:35729), 8526fd24dcd0468c8106f01fb5cb81e4 (127.4.177.65:36481)
I20260808 21:09:50.775897 23425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d65af3d683e4c89a066ab440e9a296d" candidate_uuid: "6d7e3464065d43e2b30b316b932acf20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8526fd24dcd0468c8106f01fb5cb81e4"
I20260808 21:09:50.775959 23425 raft_consensus.cc:3037] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:50.776671 23425 raft_consensus.cc:2454] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e3464065d43e2b30b316b932acf20 in term 1.
I20260808 21:09:50.776798 23710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d65af3d683e4c89a066ab440e9a296d" candidate_uuid: "6d7e3464065d43e2b30b316b932acf20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a"
I20260808 21:09:50.776829 23497 leader_election.cc:302] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d7e3464065d43e2b30b316b932acf20, 8526fd24dcd0468c8106f01fb5cb81e4; no voters:
I20260808 21:09:50.776870 23710 raft_consensus.cc:3037] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:50.776948 23784 raft_consensus.cc:2781] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:50.777107 23784 raft_consensus.cc:686] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 1 LEADER]: Becoming Leader. State: Replica: 6d7e3464065d43e2b30b316b932acf20, State: Running, Role: LEADER
I20260808 21:09:50.777200 23784 consensus_queue.cc:237] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } }
I20260808 21:09:50.777781 23710 raft_consensus.cc:2454] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7e3464065d43e2b30b316b932acf20 in term 1.
I20260808 21:09:50.778031 23275 catalog_manager.cc:5495] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d7e3464065d43e2b30b316b932acf20 (127.4.177.66). New cstate: current_term: 1 leader_uuid: "6d7e3464065d43e2b30b316b932acf20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d7e3464065d43e2b30b316b932acf20" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 38673 } health_report { overall_health: HEALTHY } } }
I20260808 21:09:50.840739 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.838317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.4.177.66:38673)
0808 21:09:50.838381 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:50.838383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:50.838392 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:50.838437 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:50.838438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:50.838454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:50.838534 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:50.838665 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.839168 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.839952 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:50.840152 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:50.840231 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:50.840398 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:50.840494 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:50.840595 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:50.840614 (+ 19us) server_negotiation.cc:299] Negotiation successful
0808 21:09:50.840636 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:09:50.842620 23425 raft_consensus.cc:1261] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 1 FOLLOWER]: Refusing update from remote peer 6d7e3464065d43e2b30b316b932acf20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:50.842613 23710 raft_consensus.cc:1261] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 1 FOLLOWER]: Refusing update from remote peer 6d7e3464065d43e2b30b316b932acf20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:50.843003 23788 consensus_queue.cc:1035] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8526fd24dcd0468c8106f01fb5cb81e4" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:50.843163 23784 consensus_queue.cc:1035] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff62e938ad664f8f8e6d54cd3ae2425a" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:50.845734 23793 mvcc.cc:205] Tried to move back new op lower bound from 7316371008888471552 to 7316371008623923200. Current Snapshot: MvccSnapshot[applied={T|T < 7316371008888471552}]
I20260808 21:09:50.846421 23794 mvcc.cc:205] Tried to move back new op lower bound from 7316371008888471552 to 7316371008623923200. Current Snapshot: MvccSnapshot[applied={T|T < 7316371008888471552}]
I20260808 21:09:50.847920 23275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:50.848006 23275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:50.848785 23275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a614dd5716364c13987931cdffd6065f: Sending DeleteTablet for 3 replicas of tablet 3d65af3d683e4c89a066ab440e9a296d
I20260808 21:09:50.849076 23690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d65af3d683e4c89a066ab440e9a296d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:50 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54656
I20260808 21:09:50.849170 23404 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d65af3d683e4c89a066ab440e9a296d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:50 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:54038
I20260808 21:09:50.849166 23546 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d65af3d683e4c89a066ab440e9a296d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:50 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:38608
I20260808 21:09:50.849421 23803 tablet_replica.cc:330] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4: stopping tablet replica
I20260808 21:09:50.849478 23802 tablet_replica.cc:330] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: stopping tablet replica
I20260808 21:09:50.849587 23803 raft_consensus.cc:2229] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:50.849604 23802 raft_consensus.cc:2229] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:50.849704 23802 raft_consensus.cc:2258] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:50.849702 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23338
I20260808 21:09:50.849704 23803 raft_consensus.cc:2258] T 3d65af3d683e4c89a066ab440e9a296d P 8526fd24dcd0468c8106f01fb5cb81e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:50.850749 23805 tablet_replica.cc:330] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: stopping tablet replica
I20260808 21:09:50.850859 23805 raft_consensus.cc:2229] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:50.850972 23805 raft_consensus.cc:2258] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:50.851750 23805 ts_tablet_manager.cc:1905] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:50.853319 23805 ts_tablet_manager.cc:1918] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:50.853410 23805 log.cc:1196] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/3d65af3d683e4c89a066ab440e9a296d
I20260808 21:09:50.853657 23805 ts_tablet_manager.cc:1939] T 3d65af3d683e4c89a066ab440e9a296d P 6d7e3464065d43e2b30b316b932acf20: Deleting consensus metadata
I20260808 21:09:50.854056 23261 catalog_manager.cc:4841] TS 6d7e3464065d43e2b30b316b932acf20 (127.4.177.66:38673): tablet 3d65af3d683e4c89a066ab440e9a296d (table test-table [id=7d8225938f5745eeae934ea389b6775b]) successfully deleted
I20260808 21:09:50.855966 23802 ts_tablet_manager.cc:1905] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:50.857586 23802 ts_tablet_manager.cc:1918] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:50.857671 23802 log.cc:1196] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/3d65af3d683e4c89a066ab440e9a296d
I20260808 21:09:50.857949 23802 ts_tablet_manager.cc:1939] T 3d65af3d683e4c89a066ab440e9a296d P ff62e938ad664f8f8e6d54cd3ae2425a: Deleting consensus metadata
I20260808 21:09:50.858376 23259 catalog_manager.cc:4841] TS ff62e938ad664f8f8e6d54cd3ae2425a (127.4.177.67:35729): tablet 3d65af3d683e4c89a066ab440e9a296d (table test-table [id=7d8225938f5745eeae934ea389b6775b]) successfully deleted
W20260808 21:09:50.861691 23258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:36481 (error 108)
I20260808 21:09:50.862270 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:50.862134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:36481 (local address 127.0.0.1:54054)
0808 21:09:50.862192 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:50.862235 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:36481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 21:09:50.862401 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23479
W20260808 21:09:50.865309 23258 catalog_manager.cc:4584] TS 8526fd24dcd0468c8106f01fb5cb81e4 (127.4.177.65:36481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d65af3d683e4c89a066ab440e9a296d: Network error: Client connection negotiation failed: client connection to 127.4.177.65:36481: connect: Connection refused (error 111)
I20260808 21:09:50.869277 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23623
I20260808 21:09:50.875506 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23240
2026-08-08T21:09:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2364 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T21:09:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:50Z Disabled control of system clock
I20260808 21:09:50.906559 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42411
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:50.987614 23811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:50.987818 23811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:50.987838 23811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:50.988903 23811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:50.988937 23811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:50.988950 23811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:50.988962 23811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:50.988974 23811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:50.990239 23811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:50.990466 23811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:50.990737 23811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:50.995469 23817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:50.995503 23816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.995642 23811 server_base.cc:1034] running on GCE node
W20260808 21:09:50.995718 23819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:50.995916 23811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:50.996196 23811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:50.997359 23811 hybrid_clock.cc:648] HybridClock initialized: now 1786223390997341 us; error 32 us; skew 500 ppm
I20260808 21:09:50.998584 23811 webserver.cc:460] Webserver started at http://127.4.177.126:33873/ using document root <none> and password file <none>
I20260808 21:09:50.998885 23811 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:50.998956 23811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:50.999130 23811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:51.000275 23811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "8cc02c950f1f46cda8e30918181eb392"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "df9e13397089aad5d53e0d82c384a97e"
server_key_iv: "c61730211bd2394e44948181424ff8f1"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.000684 23811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "8cc02c950f1f46cda8e30918181eb392"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "df9e13397089aad5d53e0d82c384a97e"
server_key_iv: "c61730211bd2394e44948181424ff8f1"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.001938 23811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:51.002641 23826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.002825 23811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:51.002924 23811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "8cc02c950f1f46cda8e30918181eb392"
format_stamp: "Formatted at 2026-08-08 21:09:50 on dist-test-slave-chsk"
server_key: "df9e13397089aad5d53e0d82c384a97e"
server_key_iv: "c61730211bd2394e44948181424ff8f1"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.003006 23811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:51.023571 23811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:51.023892 23811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:51.024050 23811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:51.028155 23811 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42411
I20260808 21:09:51.028190 23880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42411 every 8 connection(s)
I20260808 21:09:51.028532 23811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:51.029523 23881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:51.031315 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23811
I20260808 21:09:51.031440 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:51.031682 4805 external_mini_cluster.cc:1428] Setting key f5b439135aa380ffff1427a8e9ae8354
I20260808 21:09:51.032333 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: Bootstrap starting.
I20260808 21:09:51.032970 23881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:51.033351 23881 log.cc:824] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:51.034051 23881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: No bootstrap required, opened a new log
I20260808 21:09:51.036480 23881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } }
I20260808 21:09:51.036657 23881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:51.036686 23881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc02c950f1f46cda8e30918181eb392, State: Initialized, Role: FOLLOWER
I20260808 21:09:51.036827 23881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } }
I20260808 21:09:51.036901 23881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:51.036924 23881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:51.036991 23881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:51.037384 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.032229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.4.177.126:42411)
0808 21:09:51.032436 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.032446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.032481 (+ 35us) server_negotiation.cc:407] Connection header received
0808 21:09:51.032875 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.032881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.032937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.033093 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:51.033335 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.034245 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.035340 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.035725 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.035812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.036599 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:51.036602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:51.036603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:51.036859 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:51.036916 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:51.036931 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:51.036996 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.037071 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.037239 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260808 21:09:51.037580 23881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } }
I20260808 21:09:51.037688 23881 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc02c950f1f46cda8e30918181eb392; no voters:
I20260808 21:09:51.037863 23881 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:51.037890 23887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:51.038093 23887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc02c950f1f46cda8e30918181eb392, State: Running, Role: LEADER
I20260808 21:09:51.038112 23881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:51.038187 23887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } }
I20260808 21:09:51.038533 23889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc02c950f1f46cda8e30918181eb392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } } }
I20260808 21:09:51.038522 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc02c950f1f46cda8e30918181eb392. Latest consensus state: current_term: 1 leader_uuid: "8cc02c950f1f46cda8e30918181eb392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc02c950f1f46cda8e30918181eb392" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42411 } } }
I20260808 21:09:51.038662 23889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:51.038671 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:51.039387 23895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:51.039943 23895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:51.041384 23895 catalog_manager.cc:1347] Generated new cluster ID: 00cae9bc63314174a7b5bafeb9af284e
I20260808 21:09:51.041447 23895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:51.048444 23895 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:51.048956 23895 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:51.052529 23895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: Generated new TSK 0
I20260808 21:09:51.052699 23895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:09:51.061550 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--builtin_ntp_servers=127.4.177.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:51.159986 23907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:51.160192 23907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:51.160213 23907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:51.161350 23907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:51.161391 23907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:51.161414 23907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:51.162782 23907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:51.163023 23907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:51.163379 23907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:51.168424 23915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.168476 23912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.168450 23913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:51.168946 23907 server_base.cc:1034] running on GCE node
I20260808 21:09:51.169191 23907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:51.169502 23907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:51.170671 23907 hybrid_clock.cc:648] HybridClock initialized: now 1786223391170659 us; error 30 us; skew 500 ppm
I20260808 21:09:51.172039 23907 webserver.cc:460] Webserver started at http://127.4.177.65:42623/ using document root <none> and password file <none>
I20260808 21:09:51.172308 23907 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:51.172364 23907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:51.172515 23907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:51.173601 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "5a8aacd75b484f3690b5e8456eb6f5ac"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "61a1dea2f0e97cfabdd10fd0fb82df87"
server_key_iv: "cfd77bfb411ff1f2a7c5ebaaf7aa0b1d"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.173965 23907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "5a8aacd75b484f3690b5e8456eb6f5ac"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "61a1dea2f0e97cfabdd10fd0fb82df87"
server_key_iv: "cfd77bfb411ff1f2a7c5ebaaf7aa0b1d"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.175387 23907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:51.176229 23922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.176425 23907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:51.176528 23907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "5a8aacd75b484f3690b5e8456eb6f5ac"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "61a1dea2f0e97cfabdd10fd0fb82df87"
server_key_iv: "cfd77bfb411ff1f2a7c5ebaaf7aa0b1d"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.176620 23907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:51.202572 23907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:51.202899 23907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:51.203052 23907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:51.203325 23907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:51.203672 23907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:51.203737 23907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.203780 23907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:51.203804 23907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.209884 23907 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:41187
I20260808 21:09:51.209937 24041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:41187 every 8 connection(s)
I20260808 21:09:51.210300 23907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:51.215359 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.211068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:34989 (local address 127.4.177.126:42411)
0808 21:09:51.211181 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.211184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.211394 (+ 210us) server_negotiation.cc:407] Connection header received
0808 21:09:51.211858 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.211859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.211872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.211929 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:51.212555 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.213052 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.213974 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.214155 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.214205 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.215058 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:51.215059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:51.215060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:51.215065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:51.215078 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:51.215083 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:51.215176 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.215282 (+ 106us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.215302 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 21:09:51.215562 24044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.211025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42411 (local address 127.4.177.65:34989)
0808 21:09:51.211276 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.211309 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.211778 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.211978 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.211981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.211988 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:51.212486 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.212490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.213134 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.213135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.213902 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.213904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.214230 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.214230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.214333 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.214923 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:51.214923 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:51.214951 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:51.214971 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:51.215201 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:51.215208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:51.215211 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.215272 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.215352 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":136,"threads_started":1}
I20260808 21:09:51.216355 24042 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42411
I20260808 21:09:51.216481 24042 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:51.216683 24042 heartbeater.cc:511] Master 127.4.177.126:42411 requested a full tablet report, sending...
I20260808 21:09:51.217188 23844 ts_manager.cc:194] Registered new tserver with Master: 5a8aacd75b484f3690b5e8456eb6f5ac (127.4.177.65:41187)
I20260808 21:09:51.217809 23844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.65:34989
I20260808 21:09:51.218645 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 23907
I20260808 21:09:51.218734 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:51.218899 4805 external_mini_cluster.cc:1428] Setting key 4b8bf488dac356d097fb25fad1a8f5ad
I20260808 21:09:51.219798 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--builtin_ntp_servers=127.4.177.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:51.301352 24045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:51.301553 24045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:51.301574 24045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:51.302767 24045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:51.302816 24045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:51.302845 24045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:51.304242 24045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:51.304493 24045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:51.304821 24045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:51.309888 24050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.310223 24053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.310228 24051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:51.310516 24045 server_base.cc:1034] running on GCE node
I20260808 21:09:51.310770 24045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:51.311022 24045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:51.312234 24045 hybrid_clock.cc:648] HybridClock initialized: now 1786223391312231 us; error 42 us; skew 500 ppm
I20260808 21:09:51.313676 24045 webserver.cc:460] Webserver started at http://127.4.177.66:43577/ using document root <none> and password file <none>
I20260808 21:09:51.313959 24045 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:51.314010 24045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:51.314149 24045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:51.315306 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "1ca2ffa4913c4a838894aea8d169a52c"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "5127fa9dcd9455ebb704e78fb2d5d354"
server_key_iv: "aa23bfb97b44182c94f3388c48ad0439"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.315668 24045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "1ca2ffa4913c4a838894aea8d169a52c"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "5127fa9dcd9455ebb704e78fb2d5d354"
server_key_iv: "aa23bfb97b44182c94f3388c48ad0439"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.317013 24045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:51.317768 24059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.317956 24045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:51.318060 24045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "1ca2ffa4913c4a838894aea8d169a52c"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "5127fa9dcd9455ebb704e78fb2d5d354"
server_key_iv: "aa23bfb97b44182c94f3388c48ad0439"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.318140 24045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:51.334926 24045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:51.335332 24045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:51.335494 24045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:51.335762 24045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:51.336128 24045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:51.336184 24045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.336229 24045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:51.336253 24045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.342401 24045 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:37691
I20260808 21:09:51.342445 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:37691 every 8 connection(s)
I20260808 21:09:51.342785 24045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:51.345129 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24045
I20260808 21:09:51.345242 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:51.345443 4805 external_mini_cluster.cc:1428] Setting key 7b0dd0b7e7be7fc19d2ecda598fff97e
I20260808 21:09:51.346428 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--builtin_ntp_servers=127.4.177.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 21:09:51.348318 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.343498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:57669 (local address 127.4.177.126:42411)
0808 21:09:51.343587 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.343590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.343838 (+ 248us) server_negotiation.cc:407] Connection header received
0808 21:09:51.344289 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.344290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.344301 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.344355 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:51.344997 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.345746 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.346865 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.347084 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.347192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.348031 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:51.348032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:51.348033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:51.348038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:51.348052 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:51.348056 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:51.348115 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.348212 (+ 97us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.348238 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 21:09:51.348529 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.343482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42411 (local address 127.4.177.66:57669)
0808 21:09:51.343707 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.343742 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.344207 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.344406 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.344409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.344417 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:51.344916 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.344919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.345862 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.345867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.346776 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.346778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.347186 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.347187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.347323 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.347894 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:51.347895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:51.347921 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:51.347942 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:51.348123 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:51.348128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:51.348131 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.348212 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.348309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260808 21:09:51.349040 24178 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42411
I20260808 21:09:51.349148 24178 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:51.349344 24178 heartbeater.cc:511] Master 127.4.177.126:42411 requested a full tablet report, sending...
I20260808 21:09:51.349720 23844 ts_manager.cc:194] Registered new tserver with Master: 1ca2ffa4913c4a838894aea8d169a52c (127.4.177.66:37691)
I20260808 21:09:51.350052 23844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.66:57669
W20260808 21:09:51.430936 24181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:51.431209 24181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:51.431252 24181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:51.432464 24181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:51.432528 24181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:51.432566 24181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:51.433876 24181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:51.434135 24181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:51.434437 24181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:51.439447 24186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.439458 24189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:51.439461 24187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:51.439797 24181 server_base.cc:1034] running on GCE node
I20260808 21:09:51.440001 24181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:51.440231 24181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:51.441393 24181 hybrid_clock.cc:648] HybridClock initialized: now 1786223391441382 us; error 29 us; skew 500 ppm
I20260808 21:09:51.442682 24181 webserver.cc:460] Webserver started at http://127.4.177.67:37871/ using document root <none> and password file <none>
I20260808 21:09:51.442929 24181 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:51.443001 24181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:51.443188 24181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:51.444227 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "46739c21e1fd4467a1e630835377ce8d"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "b370a3e36c2f00ba3e0d78e3a3bb1482"
server_key_iv: "69ed9c954bad638a81182b704d66f372"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.444589 24181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "46739c21e1fd4467a1e630835377ce8d"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "b370a3e36c2f00ba3e0d78e3a3bb1482"
server_key_iv: "69ed9c954bad638a81182b704d66f372"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.445969 24181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:51.446659 24195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.446828 24181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:51.446923 24181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "46739c21e1fd4467a1e630835377ce8d"
format_stamp: "Formatted at 2026-08-08 21:09:51 on dist-test-slave-chsk"
server_key: "b370a3e36c2f00ba3e0d78e3a3bb1482"
server_key_iv: "69ed9c954bad638a81182b704d66f372"
server_key_version: "encryptionkey@0"
I20260808 21:09:51.447005 24181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:51.462747 24181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:51.463078 24181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:51.463308 24181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:51.463552 24181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:51.463897 24181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:51.463953 24181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.463994 24181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:51.464025 24181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:51.469918 24181 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:36279
I20260808 21:09:51.469964 24315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:36279 every 8 connection(s)
I20260808 21:09:51.470314 24181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:51.472733 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24181
I20260808 21:09:51.472846 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:51.473034 4805 external_mini_cluster.cc:1428] Setting key 995a89c946052a90142752c989913ea8
I20260808 21:09:51.475750 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.471061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:54705 (local address 127.4.177.126:42411)
0808 21:09:51.471168 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.471170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.471408 (+ 238us) server_negotiation.cc:407] Connection header received
0808 21:09:51.471870 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.471870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.471881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.471951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:51.472613 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.473284 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.474513 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.474690 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.474743 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.475536 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:51.475537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:51.475538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:51.475542 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:51.475554 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:51.475558 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:51.475600 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.475671 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.475692 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 21:09:51.475912 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.471075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42411 (local address 127.4.177.67:54705)
0808 21:09:51.471276 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.471315 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.471790 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.471998 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.472000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.472008 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:51.472532 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.472537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.473396 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.473398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.474442 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.474445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.474780 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.474781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.474889 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.475405 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:51.475405 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:51.475431 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:51.475457 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:51.475600 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:51.475602 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:51.475604 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.475657 (+ 53us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.475727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260808 21:09:51.476339 24316 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42411
I20260808 21:09:51.476418 24316 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:51.476557 24316 heartbeater.cc:511] Master 127.4.177.126:42411 requested a full tablet report, sending...
I20260808 21:09:51.476864 23844 ts_manager.cc:194] Registered new tserver with Master: 46739c21e1fd4467a1e630835377ce8d (127.4.177.67:36279)
I20260808 21:09:51.477239 23844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.67:54705
I20260808 21:09:51.484819 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:51.489259 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.486229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.4.177.126:42411)
0808 21:09:51.486285 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.486287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.486403 (+ 116us) server_negotiation.cc:407] Connection header received
0808 21:09:51.486497 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.486498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.486510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.486574 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:51.486754 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.487376 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.488113 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.488325 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.488400 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.488998 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:51.488999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:51.489000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:51.489004 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:51.489017 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:51.489021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:51.489092 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.489154 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.489174 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260808 21:09:51.490703 23844 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:51.492255 23844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:51.498428 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:36279 (local address 127.0.0.1:49164)
0808 21:09:51.494774 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.494787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.494838 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.495127 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.495128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.495130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:51.495184 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.495185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.496086 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.496087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.497296 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.497299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.498126 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.498127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.498204 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.498218 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.498290 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.498339 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":92,"threads_started":1}
I20260808 21:09:51.498430 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.4.177.67:36279)
0808 21:09:51.494901 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.494905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.494913 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:51.494948 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.494950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.494971 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.495063 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:51.495268 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.495985 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.497377 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.498039 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.498116 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.498203 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.498303 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.498350 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
I20260808 21:09:51.498587 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41187 (local address 127.0.0.1:55968)
0808 21:09:51.494493 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.494533 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.494600 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.495037 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.495041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.495044 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:51.495106 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.495107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.496495 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.496496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.497496 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.497497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.498362 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.498364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.498429 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.498433 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.498465 (+ 32us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.498509 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260808 21:09:51.498903 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.4.177.65:41187)
0808 21:09:51.494853 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.494857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.494868 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:09:51.494909 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.494912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.494928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.495012 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:51.495301 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.496405 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.497571 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.498264 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.498658 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.498724 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.498744 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.498771 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260808 21:09:51.499552 24247 tablet_service.cc:1467] Processing CreateTablet for tablet 38092f373f504e41bd8e377e37c12bc6 (DEFAULT_TABLE table=test-table [id=a7e8826dc7f54d8ea819116859cd0217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:51.499611 23972 tablet_service.cc:1467] Processing CreateTablet for tablet 38092f373f504e41bd8e377e37c12bc6 (DEFAULT_TABLE table=test-table [id=a7e8826dc7f54d8ea819116859cd0217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:51.499954 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38092f373f504e41bd8e377e37c12bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:51.499995 23972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38092f373f504e41bd8e377e37c12bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:51.502532 24333 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Bootstrap starting.
I20260808 21:09:51.503144 24334 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: Bootstrap starting.
I20260808 21:09:51.503273 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37691 (local address 127.0.0.1:60426)
0808 21:09:51.494821 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.494834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.494895 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.495179 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.495180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.495183 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:51.495235 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.495237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.497564 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.497565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.498405 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.498408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.502946 (+ 4538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.502947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.503018 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.503021 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.503093 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.503166 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":81,"threads_started":1}
I20260808 21:09:51.503299 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.494623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.4.177.66:37691)
0808 21:09:51.494816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.494820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.494957 (+ 137us) server_negotiation.cc:407] Connection header received
0808 21:09:51.494998 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.495000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.495014 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.495079 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:51.495318 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.496395 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.498500 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.502872 (+ 4372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.502938 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.502995 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.503133 (+ 138us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.503204 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260808 21:09:51.503317 24333 tablet_bootstrap.cc:654] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:51.503644 24333 log.cc:824] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:51.503907 24334 tablet_bootstrap.cc:654] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:51.504217 24334 log.cc:824] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:51.504307 24109 tablet_service.cc:1467] Processing CreateTablet for tablet 38092f373f504e41bd8e377e37c12bc6 (DEFAULT_TABLE table=test-table [id=a7e8826dc7f54d8ea819116859cd0217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:51.504465 24333 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: No bootstrap required, opened a new log
I20260808 21:09:51.504583 24333 ts_tablet_manager.cc:1397] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:09:51.504671 24109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38092f373f504e41bd8e377e37c12bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:51.505008 24334 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: No bootstrap required, opened a new log
I20260808 21:09:51.505065 24334 ts_tablet_manager.cc:1397] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 21:09:51.506696 24333 raft_consensus.cc:348] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.506810 24333 raft_consensus.cc:374] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:51.506831 24333 raft_consensus.cc:729] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46739c21e1fd4467a1e630835377ce8d, State: Initialized, Role: FOLLOWER
I20260808 21:09:51.506945 24333 consensus_queue.cc:260] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.507218 24316 heartbeater.cc:503] Master 127.4.177.126:42411 was elected leader, sending a full tablet report...
I20260808 21:09:51.507452 24333 ts_tablet_manager.cc:1428] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 21:09:51.507633 24334 raft_consensus.cc:348] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.507784 24334 raft_consensus.cc:374] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:51.507820 24334 raft_consensus.cc:729] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8aacd75b484f3690b5e8456eb6f5ac, State: Initialized, Role: FOLLOWER
I20260808 21:09:51.507931 24334 consensus_queue.cc:260] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.508136 24338 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Bootstrap starting.
I20260808 21:09:51.508196 24334 ts_tablet_manager.cc:1428] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:09:51.508350 24042 heartbeater.cc:503] Master 127.4.177.126:42411 was elected leader, sending a full tablet report...
I20260808 21:09:51.508782 24338 tablet_bootstrap.cc:654] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:51.509136 24338 log.cc:824] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:51.509781 24338 tablet_bootstrap.cc:492] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: No bootstrap required, opened a new log
I20260808 21:09:51.509833 24338 ts_tablet_manager.cc:1397] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:51.511622 24338 raft_consensus.cc:348] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.511739 24338 raft_consensus.cc:374] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:51.511759 24338 raft_consensus.cc:729] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2ffa4913c4a838894aea8d169a52c, State: Initialized, Role: FOLLOWER
I20260808 21:09:51.511916 24338 consensus_queue.cc:260] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.512172 24338 ts_tablet_manager.cc:1428] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:09:51.512199 24178 heartbeater.cc:503] Master 127.4.177.126:42411 was elected leader, sending a full tablet report...
W20260808 21:09:51.593330 24179 tablet.cc:2369] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:51.594530 24337 raft_consensus.cc:482] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:51.594628 24337 raft_consensus.cc:504] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.595036 24337 leader_election.cc:288] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a8aacd75b484f3690b5e8456eb6f5ac (127.4.177.65:41187), 1ca2ffa4913c4a838894aea8d169a52c (127.4.177.66:37691)
I20260808 21:09:51.597813 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.595336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:58645 (local address 127.4.177.66:37691)
0808 21:09:51.595394 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.595397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.595407 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:51.595447 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.595448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.595456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.595531 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:51.595696 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.596165 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597084 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.597449 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597546 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.597555 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.597707 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.597749 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260808 21:09:51.597813 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.595275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:37691 (local address 127.4.177.67:58645)
0808 21:09:51.595326 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.595340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.595385 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.595536 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.595537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.595539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:51.595623 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.595625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.596247 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.596249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.596987 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.596990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.597547 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.597627 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.597630 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.597688 (+ 58us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.597732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260808 21:09:51.598135 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.595328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:56637 (local address 127.4.177.65:41187)
0808 21:09:51.595387 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.595390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.595692 (+ 302us) server_negotiation.cc:407] Connection header received
0808 21:09:51.595757 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.595758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.595771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.595840 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:51.596006 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.596479 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597395 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.597769 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.597866 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.598019 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.598063 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 21:09:51.598165 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38092f373f504e41bd8e377e37c12bc6" candidate_uuid: "46739c21e1fd4467a1e630835377ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2ffa4913c4a838894aea8d169a52c" is_pre_election: true
I20260808 21:09:51.598344 24130 raft_consensus.cc:2454] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46739c21e1fd4467a1e630835377ce8d in term 0.
I20260808 21:09:51.598378 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.595274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41187 (local address 127.4.177.67:56637)
0808 21:09:51.595613 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.595622 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:51.595695 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:51.595843 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:51.595843 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:51.595845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:51.595927 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.595928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.596573 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.596575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.597301 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:51.597303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.597855 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.597856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:51.597934 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.597938 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:51.598221 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 21:09:51.598270 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":159,"thread_start_us":108,"threads_started":1}
I20260808 21:09:51.598606 24198 leader_election.cc:302] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2ffa4913c4a838894aea8d169a52c, 46739c21e1fd4467a1e630835377ce8d; no voters:
I20260808 21:09:51.598747 24337 raft_consensus.cc:2781] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:51.598807 24337 raft_consensus.cc:482] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:51.598766 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38092f373f504e41bd8e377e37c12bc6" candidate_uuid: "46739c21e1fd4467a1e630835377ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" is_pre_election: true
I20260808 21:09:51.598832 24337 raft_consensus.cc:3037] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:51.598903 23992 raft_consensus.cc:2454] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46739c21e1fd4467a1e630835377ce8d in term 0.
I20260808 21:09:51.599720 24337 raft_consensus.cc:504] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.599869 24337 leader_election.cc:288] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [CANDIDATE]: Term 1 election: Requested vote from peers 5a8aacd75b484f3690b5e8456eb6f5ac (127.4.177.65:41187), 1ca2ffa4913c4a838894aea8d169a52c (127.4.177.66:37691)
I20260808 21:09:51.600033 24130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38092f373f504e41bd8e377e37c12bc6" candidate_uuid: "46739c21e1fd4467a1e630835377ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2ffa4913c4a838894aea8d169a52c"
I20260808 21:09:51.600034 23992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38092f373f504e41bd8e377e37c12bc6" candidate_uuid: "46739c21e1fd4467a1e630835377ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac"
I20260808 21:09:51.600113 23992 raft_consensus.cc:3037] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:51.600113 24130 raft_consensus.cc:3037] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:51.601016 24130 raft_consensus.cc:2454] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46739c21e1fd4467a1e630835377ce8d in term 1.
I20260808 21:09:51.601016 23992 raft_consensus.cc:2454] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46739c21e1fd4467a1e630835377ce8d in term 1.
I20260808 21:09:51.601184 24198 leader_election.cc:302] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2ffa4913c4a838894aea8d169a52c, 46739c21e1fd4467a1e630835377ce8d; no voters:
I20260808 21:09:51.601302 24337 raft_consensus.cc:2781] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:51.601444 24337 raft_consensus.cc:686] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 1 LEADER]: Becoming Leader. State: Replica: 46739c21e1fd4467a1e630835377ce8d, State: Running, Role: LEADER
I20260808 21:09:51.601531 24337 consensus_queue.cc:237] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } }
I20260808 21:09:51.602435 23844 catalog_manager.cc:5495] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 46739c21e1fd4467a1e630835377ce8d (127.4.177.67). New cstate: current_term: 1 leader_uuid: "46739c21e1fd4467a1e630835377ce8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46739c21e1fd4467a1e630835377ce8d" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 36279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 } health_report { overall_health: UNKNOWN } } }
I20260808 21:09:51.615929 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.613585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.4.177.67:36279)
0808 21:09:51.613647 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:51.613650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:51.613657 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:09:51.613692 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:51.613692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:51.613709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:51.613796 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:51.613928 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.614378 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.615157 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:51.615357 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:51.615432 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:51.615598 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:51.615697 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:51.615797 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:51.615817 (+ 20us) server_negotiation.cc:299] Negotiation successful
0808 21:09:51.615833 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:09:51.617719 23992 raft_consensus.cc:1261] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 1 FOLLOWER]: Refusing update from remote peer 46739c21e1fd4467a1e630835377ce8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:51.617720 24130 raft_consensus.cc:1261] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 1 FOLLOWER]: Refusing update from remote peer 46739c21e1fd4467a1e630835377ce8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:51.618062 24337 consensus_queue.cc:1035] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a8aacd75b484f3690b5e8456eb6f5ac" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:51.618209 24343 consensus_queue.cc:1035] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca2ffa4913c4a838894aea8d169a52c" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:51.620584 24346 mvcc.cc:205] Tried to move back new op lower bound from 7316371012063440896 to 7316371012000579584. Current Snapshot: MvccSnapshot[applied={T|T < 7316371012063440896}]
I20260808 21:09:51.620959 24347 mvcc.cc:205] Tried to move back new op lower bound from 7316371012063440896 to 7316371012000579584. Current Snapshot: MvccSnapshot[applied={T|T < 7316371012063440896}]
I20260808 21:09:51.623896 23844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:51.623996 23844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:51.624747 23844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc02c950f1f46cda8e30918181eb392: Sending DeleteTablet for 3 replicas of tablet 38092f373f504e41bd8e377e37c12bc6
I20260808 21:09:51.625059 23972 tablet_service.cc:1514] Processing DeleteTablet for tablet 38092f373f504e41bd8e377e37c12bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:51 UTC) from {username='slave'} at 127.0.0.1:55968
I20260808 21:09:51.625106 24109 tablet_service.cc:1514] Processing DeleteTablet for tablet 38092f373f504e41bd8e377e37c12bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:51 UTC) from {username='slave'} at 127.0.0.1:60426
I20260808 21:09:51.625305 24355 tablet_replica.cc:330] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac: stopping tablet replica
I20260808 21:09:51.625329 24356 tablet_replica.cc:330] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: stopping tablet replica
I20260808 21:09:51.625407 24355 raft_consensus.cc:2229] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:51.625420 24356 raft_consensus.cc:2229] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:51.625502 24355 raft_consensus.cc:2258] T 38092f373f504e41bd8e377e37c12bc6 P 5a8aacd75b484f3690b5e8456eb6f5ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:51.625514 24356 raft_consensus.cc:2258] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:51.625510 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23907
I20260808 21:09:51.629792 24247 tablet_service.cc:1514] Processing DeleteTablet for tablet 38092f373f504e41bd8e377e37c12bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:51 UTC) from {username='slave'} at 127.0.0.1:49164
I20260808 21:09:51.630023 24357 tablet_replica.cc:330] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: stopping tablet replica
I20260808 21:09:51.630146 24357 raft_consensus.cc:2229] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:51.630265 24357 raft_consensus.cc:2258] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:51.630683 24357 ts_tablet_manager.cc:1905] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:51.630920 24356 ts_tablet_manager.cc:1905] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:51.632509 24357 ts_tablet_manager.cc:1918] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:51.632606 24357 log.cc:1196] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/38092f373f504e41bd8e377e37c12bc6
I20260808 21:09:51.632879 24357 ts_tablet_manager.cc:1939] T 38092f373f504e41bd8e377e37c12bc6 P 46739c21e1fd4467a1e630835377ce8d: Deleting consensus metadata
I20260808 21:09:51.632959 24356 ts_tablet_manager.cc:1918] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:51.633039 24356 log.cc:1196] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/38092f373f504e41bd8e377e37c12bc6
I20260808 21:09:51.633292 24356 ts_tablet_manager.cc:1939] T 38092f373f504e41bd8e377e37c12bc6 P 1ca2ffa4913c4a838894aea8d169a52c: Deleting consensus metadata
I20260808 21:09:51.633343 23829 catalog_manager.cc:4841] TS 46739c21e1fd4467a1e630835377ce8d (127.4.177.67:36279): tablet 38092f373f504e41bd8e377e37c12bc6 (table test-table [id=a7e8826dc7f54d8ea819116859cd0217]) successfully deleted
I20260808 21:09:51.633895 23828 catalog_manager.cc:4841] TS 1ca2ffa4913c4a838894aea8d169a52c (127.4.177.66:37691): tablet 38092f373f504e41bd8e377e37c12bc6 (table test-table [id=a7e8826dc7f54d8ea819116859cd0217]) successfully deleted
W20260808 21:09:51.634891 23830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:41187 (error 108)
I20260808 21:09:51.635195 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24045
I20260808 21:09:51.636194 24328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:51.635314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:41187 (local address 127.0.0.1:55982)
0808 21:09:51.636116 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:51.636156 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:41187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782}
W20260808 21:09:51.639391 23830 catalog_manager.cc:4584] TS 5a8aacd75b484f3690b5e8456eb6f5ac (127.4.177.65:41187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38092f373f504e41bd8e377e37c12bc6: Network error: Client connection negotiation failed: client connection to 127.4.177.65:41187: connect: Connection refused (error 111)
I20260808 21:09:51.642587 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24181
I20260808 21:09:51.647773 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 23811
2026-08-08T21:09:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (769 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:51 dist-test-slave-chsk krb5kdc[24361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:51 dist-test-slave-chsk krb5kdc[24361](info): set up 2 sockets
Aug 08 21:09:51 dist-test-slave-chsk krb5kdc[24361](info): commencing operation
krb5kdc: starting...
W20260808 21:09:52.731336 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:09:52 dist-test-slave-chsk krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:09:52.856209 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:36845
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:52.939024 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:52.939309 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:52.939337 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:52.940441 24374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:52.940479 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:52.940493 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:52.940506 24374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:52.940517 24374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:52.941936 24374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:36845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:52.942142 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:52.942334 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:52.947091 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:52.947294 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:52.947196 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:52.947192 24374 server_base.cc:1034] running on GCE node
I20260808 21:09:52.947679 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:52.947955 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:52.949105 24374 hybrid_clock.cc:648] HybridClock initialized: now 1786223392949089 us; error 33 us; skew 500 ppm
Aug 08 21:09:52 dist-test-slave-chsk krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223392, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:52.951676 24374 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:09:52.952065 24374 webserver.cc:460] Webserver started at http://127.4.177.126:34715/ using document root <none> and password file <none>
I20260808 21:09:52.952275 24374 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:52.952317 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:52.952404 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:52.953399 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "efb7f66efe5b4e959cacf495aa6b13ee"
format_stamp: "Formatted at 2026-08-08 21:09:52 on dist-test-slave-chsk"
server_key: "2a449f8ecb19310ba46dfe979e4c8b70"
server_key_iv: "c2b4936176889cb1a670d2f82f588431"
server_key_version: "encryptionkey@0"
I20260808 21:09:52.953701 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "efb7f66efe5b4e959cacf495aa6b13ee"
format_stamp: "Formatted at 2026-08-08 21:09:52 on dist-test-slave-chsk"
server_key: "2a449f8ecb19310ba46dfe979e4c8b70"
server_key_iv: "c2b4936176889cb1a670d2f82f588431"
server_key_version: "encryptionkey@0"
I20260808 21:09:52.954977 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:09:52.955722 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:52.955897 24374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:52.955988 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "efb7f66efe5b4e959cacf495aa6b13ee"
format_stamp: "Formatted at 2026-08-08 21:09:52 on dist-test-slave-chsk"
server_key: "2a449f8ecb19310ba46dfe979e4c8b70"
server_key_iv: "c2b4936176889cb1a670d2f82f588431"
server_key_version: "encryptionkey@0"
I20260808 21:09:52.956045 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:52.965909 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:52.968200 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:52.968317 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:52.972322 24374 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:36845
I20260808 21:09:52.972355 24447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:36845 every 8 connection(s)
I20260808 21:09:52.972725 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:52.973632 24449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:52.975986 24449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: Bootstrap starting.
I20260808 21:09:52.976596 24449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:52.976893 24449 log.cc:824] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:52.977630 24449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: No bootstrap required, opened a new log
I20260808 21:09:52.979352 24449 raft_consensus.cc:348] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } }
I20260808 21:09:52.979507 24449 raft_consensus.cc:374] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:52.979549 24449 raft_consensus.cc:729] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7f66efe5b4e959cacf495aa6b13ee, State: Initialized, Role: FOLLOWER
I20260808 21:09:52.979714 24449 consensus_queue.cc:260] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } }
I20260808 21:09:52.979800 24449 raft_consensus.cc:388] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:52.979844 24449 raft_consensus.cc:482] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:52.979897 24449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:52.980448 24449 raft_consensus.cc:504] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } }
I20260808 21:09:52.980567 24449 leader_election.cc:302] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efb7f66efe5b4e959cacf495aa6b13ee; no voters:
I20260808 21:09:52.980763 24449 leader_election.cc:288] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:52.980791 24452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:52.980968 24452 raft_consensus.cc:686] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [term 1 LEADER]: Becoming Leader. State: Replica: efb7f66efe5b4e959cacf495aa6b13ee, State: Running, Role: LEADER
I20260808 21:09:52.981041 24449 sys_catalog.cc:564] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:52.981117 24452 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } }
I20260808 21:09:52.981302 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24374
I20260808 21:09:52.981429 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:52.981516 24454 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb7f66efe5b4e959cacf495aa6b13ee. Latest consensus state: current_term: 1 leader_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } } }
I20260808 21:09:52.981611 24454 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:52.981647 4805 external_mini_cluster.cc:1428] Setting key 006eb5a4e1331b218e47d4bdb466a15a
I20260808 21:09:52.982091 24453 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb7f66efe5b4e959cacf495aa6b13ee" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36845 } } }
I20260808 21:09:52.982234 24453 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:52.982437 24456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:52.983024 24456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:52.984602 24456 catalog_manager.cc:1347] Generated new cluster ID: 04c20899ea9d47adbcc7c26518246dd0
I20260808 21:09:52.984655 24456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:09:52 dist-test-slave-chsk krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:52.994338 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:52.982223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.4.177.126:36845)
0808 21:09:52.982395 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:52.982404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:52.982442 (+ 38us) server_negotiation.cc:407] Connection header received
0808 21:09:52.982586 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:52.982593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:52.982661 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:52.982824 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:52.983743 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:52.985122 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:52.986188 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:52.986844 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:52.986955 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:52.989822 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:52.989826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:52.989826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:52.989838 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:52.992484 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:52.992908 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:52.992911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:52.992911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:52.993030 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:52.993248 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:52.993249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:52.993249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:52.993560 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:52.993711 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:52.993826 (+ 115us) server_negotiation.cc:299] Negotiation successful
0808 21:09:52.994072 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260808 21:09:53.002498 24456 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:53.003259 24456 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:53.006227 24456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: Generated new TSK 0
I20260808 21:09:53.006405 24456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:09:53.038213 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:53.121668 24476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:53.121925 24476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:53.121968 24476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:53.123183 24476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:53.123251 24476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:53.123296 24476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:53.124761 24476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:53.125022 24476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:53.125267 24476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:53.130498 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.130645 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.130529 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.130784 24476 server_base.cc:1034] running on GCE node
I20260808 21:09:53.130995 24476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:53.131271 24476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:53.132447 24476 hybrid_clock.cc:648] HybridClock initialized: now 1786223393132436 us; error 39 us; skew 500 ppm
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:53.134958 24476 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:09:53.135497 24476 webserver.cc:460] Webserver started at http://127.4.177.65:37295/ using document root <none> and password file <none>
I20260808 21:09:53.135735 24476 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:53.135805 24476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:53.135941 24476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:53.137004 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "ea208f865cf94345a26949408a958549"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "8fd08d3bae58f8cec35b582f65e4b49e"
server_key_iv: "dc1c805675ccf4e5ea93a2a71db8a2b2"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.137379 24476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "ea208f865cf94345a26949408a958549"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "8fd08d3bae58f8cec35b582f65e4b49e"
server_key_iv: "dc1c805675ccf4e5ea93a2a71db8a2b2"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.138723 24476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:53.139505 24493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.139676 24476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.139786 24476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "ea208f865cf94345a26949408a958549"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "8fd08d3bae58f8cec35b582f65e4b49e"
server_key_iv: "dc1c805675ccf4e5ea93a2a71db8a2b2"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.139868 24476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:53.154207 24476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:53.156168 24476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:53.156338 24476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:53.156595 24476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:53.156921 24476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:53.156977 24476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.157024 24476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:53.157048 24476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.163123 24476 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:37299
I20260808 21:09:53.163198 24611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:37299 every 8 connection(s)
I20260808 21:09:53.163514 24476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:53.163969 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24476
I20260808 21:09:53.164084 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:53.164321 4805 external_mini_cluster.cc:1428] Setting key a5faa7118472d2e4e97172054fce9eb4
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:53.178565 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.164275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:42715 (local address 127.4.177.126:36845)
0808 21:09:53.164321 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.164323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.164667 (+ 344us) server_negotiation.cc:407] Connection header received
0808 21:09:53.165315 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.165316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.165327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.165368 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.166664 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.167217 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.171766 (+ 4549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.172150 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.172226 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.175291 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.175293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.175293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:53.175302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.176969 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.177534 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.177535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.177536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.177590 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.177888 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.177888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.177888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.177994 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.178044 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.178410 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.178494 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:09:53.178874 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.164277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:36845 (local address 127.4.177.65:42715)
0808 21:09:53.164549 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.164592 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.165215 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.165439 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.165444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.166298 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:53.166573 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.166579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.168058 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.168060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.171608 (+ 3548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.171612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.172248 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.172249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.172419 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.173056 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:53.173066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:53.175158 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:53.177101 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.177110 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.177112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.177453 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.177660 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.177662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.177662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.177815 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.178063 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:53.178071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:53.178257 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.178393 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.178581 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
I20260808 21:09:53.179430 24613 heartbeater.cc:347] Connected to a master server at 127.4.177.126:36845
I20260808 21:09:53.179554 24613 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:53.179790 24613 heartbeater.cc:511] Master 127.4.177.126:36845 requested a full tablet report, sending...
I20260808 21:09:53.180873 24410 ts_manager.cc:194] Registered new tserver with Master: ea208f865cf94345a26949408a958549 (127.4.177.65:37299)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
I20260808 21:09:53.183320 24410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:42715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:09:53.200385 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:53.288570 24618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:53.288775 24618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:53.288798 24618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:53.289855 24618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:53.289888 24618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:53.289913 24618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:53.291280 24618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:53.291450 24618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:53.291706 24618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:53.297020 24627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.297030 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.297226 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.297039 24618 server_base.cc:1034] running on GCE node
I20260808 21:09:53.297520 24618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:53.297760 24618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:53.298920 24618 hybrid_clock.cc:648] HybridClock initialized: now 1786223393298894 us; error 37 us; skew 500 ppm
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:53.301453 24618 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:09:53.301879 24618 webserver.cc:460] Webserver started at http://127.4.177.66:36319/ using document root <none> and password file <none>
I20260808 21:09:53.302114 24618 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:53.302181 24618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:53.302345 24618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:53.303455 24618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "7ca21744bde142d1850d9688e966406b"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "229b60bb9d798b22b3a8c1bc45fa9c86"
server_key_iv: "acc4ef0a7b75de72f6f0b1dd9f9e91fa"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.303817 24618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "7ca21744bde142d1850d9688e966406b"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "229b60bb9d798b22b3a8c1bc45fa9c86"
server_key_iv: "acc4ef0a7b75de72f6f0b1dd9f9e91fa"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.305119 24618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:09:53.305889 24634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.306069 24618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:53.306169 24618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "7ca21744bde142d1850d9688e966406b"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "229b60bb9d798b22b3a8c1bc45fa9c86"
server_key_iv: "acc4ef0a7b75de72f6f0b1dd9f9e91fa"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.306263 24618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:53.319244 24618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:53.321555 24618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:53.321720 24618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:53.321964 24618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:53.322289 24618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:53.322345 24618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.322386 24618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:53.322414 24618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.328476 24618 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:33179
I20260808 21:09:53.328558 24751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:33179 every 8 connection(s)
I20260808 21:09:53.328876 24618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:53.337592 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24618
I20260808 21:09:53.337734 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:53.337968 4805 external_mini_cluster.cc:1428] Setting key 08b14a91b753a1089982eb966fd0b6ac
I20260808 21:09:53.338406 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.329670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:37919 (local address 127.4.177.126:36845)
0808 21:09:53.329753 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.329756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.329948 (+ 192us) server_negotiation.cc:407] Connection header received
0808 21:09:53.330489 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.330490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.330504 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.330569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.331623 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.332147 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.333164 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.333387 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.333465 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.335674 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.335676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.335676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:53.335685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.336863 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.337231 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.337232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.337232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.337281 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.337582 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.337583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.337583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.337697 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.337798 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.338184 (+ 386us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.338285 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:09:53.338577 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.329613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:36845 (local address 127.4.177.66:37919)
0808 21:09:53.329818 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.329863 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.330377 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.330638 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.330643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.331320 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:53.331519 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.331524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.332250 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.332251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.333052 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.333054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.333474 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.333475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.333599 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.334094 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:53.334102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:53.335523 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:53.336935 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.336942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.336943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.337161 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.337339 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.337340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.337340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.337504 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.337784 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:53.337791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:53.337961 (+ 170us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.338137 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.338330 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260808 21:09:53.339075 24752 heartbeater.cc:347] Connected to a master server at 127.4.177.126:36845
I20260808 21:09:53.339210 24752 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:53.339401 24752 heartbeater.cc:511] Master 127.4.177.126:36845 requested a full tablet report, sending...
I20260808 21:09:53.339829 24410 ts_manager.cc:194] Registered new tserver with Master: 7ca21744bde142d1850d9688e966406b (127.4.177.66:33179)
I20260808 21:09:53.340328 24410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:37919
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:09:53.365901 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:53.447434 24757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:53.447625 24757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:53.447649 24757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:53.448693 24757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:53.448727 24757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:53.448753 24757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:53.450106 24757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:36845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:53.450299 24757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:53.450474 24757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:53.455417 24763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.455459 24762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.455600 24757 server_base.cc:1034] running on GCE node
W20260808 21:09:53.455480 24766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.455895 24757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:53.456174 24757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:53.457371 24757 hybrid_clock.cc:648] HybridClock initialized: now 1786223393457351 us; error 38 us; skew 500 ppm
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:53.460158 24757 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:09:53.460600 24757 webserver.cc:460] Webserver started at http://127.4.177.67:34377/ using document root <none> and password file <none>
I20260808 21:09:53.460858 24757 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:53.460929 24757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:53.461083 24757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:53.462157 24757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "a243f8a32bbe49639cda38a14a7270ba"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "15f4678f124621dd31b27531dc8d4858"
server_key_iv: "6a5f9319a00e130b7649f9d5a418c95f"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.462538 24757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "a243f8a32bbe49639cda38a14a7270ba"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "15f4678f124621dd31b27531dc8d4858"
server_key_iv: "6a5f9319a00e130b7649f9d5a418c95f"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.463967 24757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:53.464679 24775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.464850 24757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:53.464960 24757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "a243f8a32bbe49639cda38a14a7270ba"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "15f4678f124621dd31b27531dc8d4858"
server_key_iv: "6a5f9319a00e130b7649f9d5a418c95f"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.465034 24757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:53.475258 24757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:53.477705 24757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:53.477881 24757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:53.478132 24757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:53.478458 24757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:53.478514 24757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.478561 24757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:53.478595 24757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.485143 24757 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:38561
I20260808 21:09:53.485226 24890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:38561 every 8 connection(s)
I20260808 21:09:53.485567 24757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:53.491333 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24757
I20260808 21:09:53.491425 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:53.491631 4805 external_mini_cluster.cc:1428] Setting key 3fde4da5386c0bf71b985f1bf6a76272
Aug 08 21:09:53 dist-test-slave-chsk krb5kdc[24361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:53.495469 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.486327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:52679 (local address 127.4.177.126:36845)
0808 21:09:53.486392 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.486394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.486598 (+ 204us) server_negotiation.cc:407] Connection header received
0808 21:09:53.487121 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.487122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.487158 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.487215 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.488178 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.488681 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.489706 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.489903 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.489979 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.492417 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.492419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.492419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:53.492428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.494001 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.494432 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.494433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.494434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.494499 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.494764 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.494765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.494766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.494894 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.494971 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.495292 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.495377 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:09:53.495654 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.486323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:36845 (local address 127.4.177.67:52679)
0808 21:09:53.486474 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.486509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.487012 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.487291 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.487295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.487873 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:53.488065 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.488070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.488782 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.488783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.489594 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.489596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.489987 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.489988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.490112 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.490600 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:53.490608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:53.492336 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:53.494106 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.494115 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.494118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.494352 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.494570 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:53.494571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:53.494572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:53.494690 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:53.494961 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:53.494969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:53.495114 (+ 145us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.495250 (+ 136us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.495417 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260808 21:09:53.496142 24891 heartbeater.cc:347] Connected to a master server at 127.4.177.126:36845
I20260808 21:09:53.496249 24891 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:53.496398 24891 heartbeater.cc:511] Master 127.4.177.126:36845 requested a full tablet report, sending...
I20260808 21:09:53.496758 24410 ts_manager.cc:194] Registered new tserver with Master: a243f8a32bbe49639cda38a14a7270ba (127.4.177.67:38561)
I20260808 21:09:53.497099 24410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:52679
I20260808 21:09:53.503595 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:53.511739 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.504970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.4.177.126:36845)
0808 21:09:53.505065 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.505067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.505160 (+ 93us) server_negotiation.cc:407] Connection header received
0808 21:09:53.505242 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.505243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.505253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.505310 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.505897 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.506500 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.507285 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.507500 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.507576 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.509003 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.509005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.509005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:53.509013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.510385 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.510797 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.510798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.510798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.510859 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:53.511087 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:53.511088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:53.511088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:53.511312 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.511433 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.511562 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.511629 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 21:09:53.513489 24410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:53.515130 24410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:53.521723 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37299 (local address 127.0.0.1:38434)
0808 21:09:53.517808 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.517818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.517858 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.518205 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.518210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.518212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:53.518295 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.518297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519363 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.519364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.520533 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.520536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521381 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.521382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.521466 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.521479 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.521572 (+ 93us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.521632 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":121,"threads_started":1}
I20260808 21:09:53.521729 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.4.177.65:37299)
0808 21:09:53.518014 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.518019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.518032 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:09:53.518078 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.518080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.518095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.518190 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:53.518390 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519259 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.520636 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521283 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.521368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.521431 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.521589 (+ 158us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.521632 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260808 21:09:53.521993 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:33179 (local address 127.0.0.1:43482)
0808 21:09:53.517961 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.517969 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.518008 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.518421 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.518421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.518423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:53.518458 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.518459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519856 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.519857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.520765 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.520767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521746 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.521747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.521820 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.521828 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.521883 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.521914 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260808 21:09:53.522063 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.4.177.66:33179)
0808 21:09:53.518145 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.518155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.518166 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:09:53.518208 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.518210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.518227 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.518341 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:53.518534 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519713 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.520852 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521576 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.521651 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.521710 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.521909 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.521960 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":76,"threads_started":1}
I20260808 21:09:53.521992 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:38561 (local address 127.0.0.1:59626)
0808 21:09:53.517716 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.517768 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.517841 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.518367 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.518368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.518370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:53.518419 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.518421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519829 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.519830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.520887 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.520889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521750 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.521751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.521820 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.521826 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.521859 (+ 33us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.521904 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":4,"thread_start_us":51,"threads_started":1}
I20260808 21:09:53.522436 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.517967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.4.177.67:38561)
0808 21:09:53.518193 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.518198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.518207 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:53.518244 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.518246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.518263 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.518496 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:53.518574 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.519713 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.520970 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.521657 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.522107 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.522321 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.522343 (+ 22us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.522377 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260808 21:09:53.522886 24684 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 (DEFAULT_TABLE table=test-table [id=213ccbb4f7f1479cae45787314ce57aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:53.523329 24684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67bed924d74f29a13dbbcc5dcb5464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:53.523211 24825 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 (DEFAULT_TABLE table=test-table [id=213ccbb4f7f1479cae45787314ce57aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:53.523563 24825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67bed924d74f29a13dbbcc5dcb5464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:53.523693 24544 tablet_service.cc:1467] Processing CreateTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 (DEFAULT_TABLE table=test-table [id=213ccbb4f7f1479cae45787314ce57aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:53.523936 24544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b67bed924d74f29a13dbbcc5dcb5464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:53.527019 24909 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: Bootstrap starting.
I20260808 21:09:53.527122 24910 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Bootstrap starting.
I20260808 21:09:53.527340 24911 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Bootstrap starting.
I20260808 21:09:53.527807 24910 tablet_bootstrap.cc:654] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:53.527827 24909 tablet_bootstrap.cc:654] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:53.527976 24911 tablet_bootstrap.cc:654] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:53.528362 24910 log.cc:824] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:53.528510 24909 log.cc:824] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:53.528587 24911 log.cc:824] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:53.529282 24910 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: No bootstrap required, opened a new log
I20260808 21:09:53.529338 24911 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: No bootstrap required, opened a new log
I20260808 21:09:53.529363 24910 ts_tablet_manager.cc:1397] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:09:53.529405 24911 ts_tablet_manager.cc:1397] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:53.529479 24909 tablet_bootstrap.cc:492] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: No bootstrap required, opened a new log
I20260808 21:09:53.529551 24909 ts_tablet_manager.cc:1397] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260808 21:09:53.530922 24909 raft_consensus.cc:348] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.531100 24909 raft_consensus.cc:374] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:53.531196 24909 raft_consensus.cc:729] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea208f865cf94345a26949408a958549, State: Initialized, Role: FOLLOWER
I20260808 21:09:53.531339 24909 consensus_queue.cc:260] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.531424 24910 raft_consensus.cc:348] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.531424 24911 raft_consensus.cc:348] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.531607 24911 raft_consensus.cc:374] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:53.531607 24910 raft_consensus.cc:374] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:53.531652 24910 raft_consensus.cc:729] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a243f8a32bbe49639cda38a14a7270ba, State: Initialized, Role: FOLLOWER
I20260808 21:09:53.531652 24911 raft_consensus.cc:729] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca21744bde142d1850d9688e966406b, State: Initialized, Role: FOLLOWER
I20260808 21:09:53.531673 24909 ts_tablet_manager.cc:1428] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:53.531772 24910 consensus_queue.cc:260] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.531873 24613 heartbeater.cc:503] Master 127.4.177.126:36845 was elected leader, sending a full tablet report...
I20260808 21:09:53.532078 24910 ts_tablet_manager.cc:1428] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:09:53.532097 24891 heartbeater.cc:503] Master 127.4.177.126:36845 was elected leader, sending a full tablet report...
I20260808 21:09:53.531801 24911 consensus_queue.cc:260] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.532436 24911 ts_tablet_manager.cc:1428] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:09:53.532562 24752 heartbeater.cc:503] Master 127.4.177.126:36845 was elected leader, sending a full tablet report...
I20260808 21:09:53.558297 24917 raft_consensus.cc:482] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:53.558446 24917 raft_consensus.cc:504] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.558858 24917 leader_election.cc:288] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea208f865cf94345a26949408a958549 (127.4.177.65:37299), 7ca21744bde142d1850d9688e966406b (127.4.177.66:33179)
I20260808 21:09:53.562074 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:33179 (local address 127.4.177.67:32943)
0808 21:09:53.559104 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.559117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.559197 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.559389 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.559391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.559393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:53.559472 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.559475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.560419 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.560419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.561138 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.561140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.561758 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.561855 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.561858 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.561943 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.561987 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 21:09:53.562074 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.559199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:32943 (local address 127.4.177.66:33179)
0808 21:09:53.559249 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.559252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.559262 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:53.559293 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.559294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.559305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.559372 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:53.559549 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.560337 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561248 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.561654 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561749 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.561759 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.561950 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.561992 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:09:53.562219 24906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.559157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:49669 (local address 127.4.177.65:37299)
0808 21:09:53.559218 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.559220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.559476 (+ 256us) server_negotiation.cc:407] Connection header received
0808 21:09:53.559552 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.559553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.559563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.559622 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:53.559781 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.560457 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561378 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.561828 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.561923 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.562085 (+ 162us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.562128 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260808 21:09:53.562414 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37299 (local address 127.4.177.67:49669)
0808 21:09:53.559415 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.559425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.559486 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.559634 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.559634 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.559636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:53.559702 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.559704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.560547 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.560548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.561288 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.561290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.561915 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.561916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.561990 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.561994 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.562081 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.562120 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":133,"thread_start_us":138,"threads_started":1}
I20260808 21:09:53.562487 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67bed924d74f29a13dbbcc5dcb5464" candidate_uuid: "a243f8a32bbe49639cda38a14a7270ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca21744bde142d1850d9688e966406b" is_pre_election: true
I20260808 21:09:53.562661 24704 raft_consensus.cc:2454] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a243f8a32bbe49639cda38a14a7270ba in term 0.
I20260808 21:09:53.562770 24564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67bed924d74f29a13dbbcc5dcb5464" candidate_uuid: "a243f8a32bbe49639cda38a14a7270ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea208f865cf94345a26949408a958549" is_pre_election: true
I20260808 21:09:53.562930 24564 raft_consensus.cc:2454] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a243f8a32bbe49639cda38a14a7270ba in term 0.
I20260808 21:09:53.562933 24777 leader_election.cc:302] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ca21744bde142d1850d9688e966406b, a243f8a32bbe49639cda38a14a7270ba; no voters:
I20260808 21:09:53.563066 24917 raft_consensus.cc:2781] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:53.563120 24917 raft_consensus.cc:482] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:53.563174 24917 raft_consensus.cc:3037] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:53.564002 24917 raft_consensus.cc:504] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.564158 24917 leader_election.cc:288] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [CANDIDATE]: Term 1 election: Requested vote from peers ea208f865cf94345a26949408a958549 (127.4.177.65:37299), 7ca21744bde142d1850d9688e966406b (127.4.177.66:33179)
I20260808 21:09:53.564284 24564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67bed924d74f29a13dbbcc5dcb5464" candidate_uuid: "a243f8a32bbe49639cda38a14a7270ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea208f865cf94345a26949408a958549"
I20260808 21:09:53.564314 24704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b67bed924d74f29a13dbbcc5dcb5464" candidate_uuid: "a243f8a32bbe49639cda38a14a7270ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca21744bde142d1850d9688e966406b"
I20260808 21:09:53.564385 24704 raft_consensus.cc:3037] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:53.564374 24564 raft_consensus.cc:3037] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:53.565297 24704 raft_consensus.cc:2454] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a243f8a32bbe49639cda38a14a7270ba in term 1.
I20260808 21:09:53.565296 24564 raft_consensus.cc:2454] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a243f8a32bbe49639cda38a14a7270ba in term 1.
I20260808 21:09:53.565464 24777 leader_election.cc:302] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ca21744bde142d1850d9688e966406b, a243f8a32bbe49639cda38a14a7270ba; no voters:
I20260808 21:09:53.565567 24917 raft_consensus.cc:2781] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:53.565747 24917 raft_consensus.cc:686] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 1 LEADER]: Becoming Leader. State: Replica: a243f8a32bbe49639cda38a14a7270ba, State: Running, Role: LEADER
I20260808 21:09:53.565883 24917 consensus_queue.cc:237] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } }
I20260808 21:09:53.566715 24408 catalog_manager.cc:5495] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba reported cstate change: term changed from 0 to 1, leader changed from <none> to a243f8a32bbe49639cda38a14a7270ba (127.4.177.67). New cstate: current_term: 1 leader_uuid: "a243f8a32bbe49639cda38a14a7270ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a243f8a32bbe49639cda38a14a7270ba" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 } health_report { overall_health: UNKNOWN } } }
W20260808 21:09:53.579389 24753 tablet.cc:2369] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:53.579658 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.577405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.4.177.67:38561)
0808 21:09:53.577471 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.577473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.577481 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:53.577508 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.577509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.577528 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.577587 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:53.577721 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.578162 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.578852 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.579026 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.579074 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.579286 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:53.579370 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:53.579447 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.579540 (+ 93us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.579553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260808 21:09:53.581405 24704 raft_consensus.cc:1261] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 1 FOLLOWER]: Refusing update from remote peer a243f8a32bbe49639cda38a14a7270ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:53.581418 24564 raft_consensus.cc:1261] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 1 FOLLOWER]: Refusing update from remote peer a243f8a32bbe49639cda38a14a7270ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:53.581754 24922 consensus_queue.cc:1035] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ca21744bde142d1850d9688e966406b" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:53.581908 24917 consensus_queue.cc:1035] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea208f865cf94345a26949408a958549" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:53.584471 24927 mvcc.cc:205] Tried to move back new op lower bound from 7316371020106702848 to 7316371020046503936. Current Snapshot: MvccSnapshot[applied={T|T < 7316371020106702848 or (T in {7316371020106702848})}]
I20260808 21:09:53.584591 24929 mvcc.cc:205] Tried to move back new op lower bound from 7316371020106702848 to 7316371020046503936. Current Snapshot: MvccSnapshot[applied={T|T < 7316371020106702848}]
I20260808 21:09:53.584718 24928 mvcc.cc:205] Tried to move back new op lower bound from 7316371020106702848 to 7316371020046503936. Current Snapshot: MvccSnapshot[applied={T|T < 7316371020106702848}]
I20260808 21:09:53.586350 24408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:53.586429 24408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:53.587222 24408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efb7f66efe5b4e959cacf495aa6b13ee: Sending DeleteTablet for 3 replicas of tablet 4b67bed924d74f29a13dbbcc5dcb5464
I20260808 21:09:53.587564 24825 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:53 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:59626
I20260808 21:09:53.587625 24544 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:53 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:38434
I20260808 21:09:53.587828 24938 tablet_replica.cc:330] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549: stopping tablet replica
I20260808 21:09:53.587827 24684 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b67bed924d74f29a13dbbcc5dcb5464 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:53 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:43482
I20260808 21:09:53.587939 24938 raft_consensus.cc:2229] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:53.588039 24938 raft_consensus.cc:2258] T 4b67bed924d74f29a13dbbcc5dcb5464 P ea208f865cf94345a26949408a958549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:53.588060 24939 tablet_replica.cc:330] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: stopping tablet replica
I20260808 21:09:53.588157 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24476
I20260808 21:09:53.588197 24937 tablet_replica.cc:330] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: stopping tablet replica
I20260808 21:09:53.588315 24939 raft_consensus.cc:2229] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:53.588464 24939 raft_consensus.cc:2258] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:53.589946 24939 ts_tablet_manager.cc:1905] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:53.590519 24937 raft_consensus.cc:2229] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:53.590627 24937 raft_consensus.cc:2258] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:53.592103 24939 ts_tablet_manager.cc:1918] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:53.592221 24939 log.cc:1196] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/4b67bed924d74f29a13dbbcc5dcb5464
I20260808 21:09:53.592526 24939 ts_tablet_manager.cc:1939] T 4b67bed924d74f29a13dbbcc5dcb5464 P 7ca21744bde142d1850d9688e966406b: Deleting consensus metadata
I20260808 21:09:53.592984 24391 catalog_manager.cc:4841] TS 7ca21744bde142d1850d9688e966406b (127.4.177.66:33179): tablet 4b67bed924d74f29a13dbbcc5dcb5464 (table test-table [id=213ccbb4f7f1479cae45787314ce57aa]) successfully deleted
I20260808 21:09:53.595635 24937 ts_tablet_manager.cc:1905] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:53.597549 24937 ts_tablet_manager.cc:1918] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:53.597640 24937 log.cc:1196] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/4b67bed924d74f29a13dbbcc5dcb5464
I20260808 21:09:53.597944 24937 ts_tablet_manager.cc:1939] T 4b67bed924d74f29a13dbbcc5dcb5464 P a243f8a32bbe49639cda38a14a7270ba: Deleting consensus metadata
I20260808 21:09:53.598367 24390 catalog_manager.cc:4841] TS a243f8a32bbe49639cda38a14a7270ba (127.4.177.67:38561): tablet 4b67bed924d74f29a13dbbcc5dcb5464 (table test-table [id=213ccbb4f7f1479cae45787314ce57aa]) successfully deleted
W20260808 21:09:53.599871 24393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:37299 (error 108)
I20260808 21:09:53.600148 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24618
I20260808 21:09:53.601939 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.601761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37299 (local address 127.0.0.1:38450)
0808 21:09:53.601835 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.601892 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 21:09:53.602090 24393 catalog_manager.cc:4584] TS ea208f865cf94345a26949408a958549 (127.4.177.65:37299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b67bed924d74f29a13dbbcc5dcb5464: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37299: connect: Connection refused (error 111)
I20260808 21:09:53.612524 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24757
I20260808 21:09:53.618742 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24374
2026-08-08T21:09:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T21:09:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:53Z Disabled control of system clock
I20260808 21:09:53.649076 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42859
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:53.733119 24944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:53.733493 24944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:53.733531 24944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:53.735613 24944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:53.735668 24944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:53.735683 24944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:53.735703 24944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:53.735716 24944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:53.737273 24944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:53.737504 24944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:53.737783 24944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:53.742736 24950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.742784 24952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.742951 24944 server_base.cc:1034] running on GCE node
W20260808 21:09:53.742751 24949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.743314 24944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:53.743731 24944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:53.744899 24944 hybrid_clock.cc:648] HybridClock initialized: now 1786223393744882 us; error 48 us; skew 500 ppm
I20260808 21:09:53.752415 24944 webserver.cc:460] Webserver started at http://127.4.177.126:36453/ using document root <none> and password file <none>
I20260808 21:09:53.752737 24944 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:53.752815 24944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:53.752976 24944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:53.754109 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b7bf20de340d48a4a215f81b77f7ddc3"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "67f101ce740516bd9df44ccac820f1f3"
server_key_iv: "92a6cfec09ac9a98957a02dbf62ce568"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.754479 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b7bf20de340d48a4a215f81b77f7ddc3"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "67f101ce740516bd9df44ccac820f1f3"
server_key_iv: "92a6cfec09ac9a98957a02dbf62ce568"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.755955 24944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:53.756800 24959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.757117 24944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:53.757226 24944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b7bf20de340d48a4a215f81b77f7ddc3"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "67f101ce740516bd9df44ccac820f1f3"
server_key_iv: "92a6cfec09ac9a98957a02dbf62ce568"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.757306 24944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:53.765363 24944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:53.765712 24944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:53.765870 24944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:53.770198 24944 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42859
I20260808 21:09:53.770236 25012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42859 every 8 connection(s)
I20260808 21:09:53.770591 24944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:53.771664 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:53.773912 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 24944
I20260808 21:09:53.774042 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:53.774138 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: Bootstrap starting.
I20260808 21:09:53.774271 4805 external_mini_cluster.cc:1428] Setting key 4ddb2be45e2f3c97b7de66e0e20adbd9
I20260808 21:09:53.774971 25013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:53.775385 25013 log.cc:824] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:53.776208 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: No bootstrap required, opened a new log
I20260808 21:09:53.778571 25013 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } }
I20260808 21:09:53.778748 25013 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:53.778779 25013 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7bf20de340d48a4a215f81b77f7ddc3, State: Initialized, Role: FOLLOWER
I20260808 21:09:53.778909 25013 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } }
I20260808 21:09:53.778982 25013 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:53.779011 25013 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:53.779053 25013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:53.779850 25013 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } }
I20260808 21:09:53.779986 25013 leader_election.cc:302] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7bf20de340d48a4a215f81b77f7ddc3; no voters:
I20260808 21:09:53.780212 25013 leader_election.cc:288] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:53.780275 25018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:53.780434 25018 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [term 1 LEADER]: Becoming Leader. State: Replica: b7bf20de340d48a4a215f81b77f7ddc3, State: Running, Role: LEADER
I20260808 21:09:53.780534 25018 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } }
I20260808 21:09:53.780550 25013 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:53.781091 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.774891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.4.177.126:42859)
0808 21:09:53.775163 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.775174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.775217 (+ 43us) server_negotiation.cc:407] Connection header received
0808 21:09:53.775776 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.775783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.775878 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.776028 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.776245 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.777685 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.778558 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.779196 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.779336 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.780082 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.780087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.780088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:53.780388 (+ 300us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.780433 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:53.780450 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.780804 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.780835 (+ 31us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.780919 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260808 21:09:53.781719 25019 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7bf20de340d48a4a215f81b77f7ddc3. Latest consensus state: current_term: 1 leader_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } } }
I20260808 21:09:53.781811 25019 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:53.782009 25020 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7bf20de340d48a4a215f81b77f7ddc3" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42859 } } }
I20260808 21:09:53.782105 25020 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3 [sys.catalog]: This master's current role is: LEADER
W20260808 21:09:53.782630 25034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:09:53.782675 25034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:09:53.782754 25030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:53.782951 25030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:53.787601 25030 catalog_manager.cc:1347] Generated new cluster ID: be7bc4aab0d641a195c2496cae8343f1
I20260808 21:09:53.787657 25030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:53.799396 25030 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:53.800148 25030 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:53.807178 25030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: Generated new TSK 0
I20260808 21:09:53.807384 25030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:09:53.816335 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--builtin_ntp_servers=127.4.177.84:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:53.900697 25038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:53.900918 25038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:53.900942 25038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:53.902218 25038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:53.902266 25038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:53.902292 25038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:53.903761 25038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:53.904043 25038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:53.904376 25038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:53.909703 25048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:53.909714 25046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.909751 25038 server_base.cc:1034] running on GCE node
W20260808 21:09:53.909724 25045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:53.910136 25038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:53.910403 25038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:53.911568 25038 hybrid_clock.cc:648] HybridClock initialized: now 1786223393911549 us; error 32 us; skew 500 ppm
I20260808 21:09:53.912840 25038 webserver.cc:460] Webserver started at http://127.4.177.65:34103/ using document root <none> and password file <none>
I20260808 21:09:53.913105 25038 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:53.913179 25038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:53.913339 25038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:53.914433 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "87c6aae214e54a10a67177f6c482160f"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "28ff730173dfbd1418f1b14396290f07"
server_key_iv: "654a3f91a90a79c775ffe7171a900d85"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.914808 25038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "87c6aae214e54a10a67177f6c482160f"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "28ff730173dfbd1418f1b14396290f07"
server_key_iv: "654a3f91a90a79c775ffe7171a900d85"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.916244 25038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:53.917037 25054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.917230 25038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:53.917340 25038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "87c6aae214e54a10a67177f6c482160f"
format_stamp: "Formatted at 2026-08-08 21:09:53 on dist-test-slave-chsk"
server_key: "28ff730173dfbd1418f1b14396290f07"
server_key_iv: "654a3f91a90a79c775ffe7171a900d85"
server_key_version: "encryptionkey@0"
I20260808 21:09:53.917418 25038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:53.931277 25038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:53.931620 25038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:53.931788 25038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:53.932044 25038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:53.932400 25038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:53.932458 25038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.932494 25038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:53.932542 25038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:53.938510 25038 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:39327
I20260808 21:09:53.938588 25171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:39327 every 8 connection(s)
I20260808 21:09:53.938978 25038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:53.942087 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25038
I20260808 21:09:53.942215 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:53.942456 4805 external_mini_cluster.cc:1428] Setting key 02d5592b59f5973e32db9b69bc03252d
I20260808 21:09:53.943528 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--builtin_ntp_servers=127.4.177.84:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 21:09:53.945129 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.939983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:46425 (local address 127.4.177.126:42859)
0808 21:09:53.940073 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:53.940075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:53.940304 (+ 229us) server_negotiation.cc:407] Connection header received
0808 21:09:53.940862 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:53.940863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:53.940874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:53.940924 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:53.941606 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.942114 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.943627 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.943878 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.943937 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.944893 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:53.944894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:53.944895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:53.944899 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:53.944913 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:53.944916 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:53.944955 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:53.945043 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 21:09:53.945067 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 21:09:53.945436 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:53.939955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42859 (local address 127.4.177.65:46425)
0808 21:09:53.940169 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:53.940215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:53.940769 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:53.940988 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:53.940992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:53.941000 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:53.941530 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.941535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.942247 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.942249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.943526 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:53.943530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:53.943970 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:53.943971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:53.944144 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:53.944732 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:53.944733 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:53.944772 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:53.944802 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:53.944967 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:53.944971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:53.944974 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:53.945040 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 21:09:53.945159 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
I20260808 21:09:53.946291 25172 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42859
I20260808 21:09:53.946417 25172 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:53.946638 25172 heartbeater.cc:511] Master 127.4.177.126:42859 requested a full tablet report, sending...
I20260808 21:09:53.947283 24977 ts_manager.cc:194] Registered new tserver with Master: 87c6aae214e54a10a67177f6c482160f (127.4.177.65:39327)
I20260808 21:09:53.948276 24977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.65:46425
W20260808 21:09:54.027724 25175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:54.027936 25175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:54.027957 25175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:54.029040 25175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:54.029073 25175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:54.029098 25175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:54.030427 25175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:54.030622 25175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:54.030877 25175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:54.035754 25183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:54.035791 25180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:54.035917 25175 server_base.cc:1034] running on GCE node
W20260808 21:09:54.036038 25181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:54.036262 25175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:54.036567 25175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:54.037744 25175 hybrid_clock.cc:648] HybridClock initialized: now 1786223394037726 us; error 48 us; skew 500 ppm
I20260808 21:09:54.039005 25175 webserver.cc:460] Webserver started at http://127.4.177.66:38455/ using document root <none> and password file <none>
I20260808 21:09:54.039335 25175 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:54.039386 25175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:54.039579 25175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:54.040726 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "bb378b093ad44c979509ee2e8f66f20f"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "42631e19f36f945a5af50534efb7234a"
server_key_iv: "10ae37962fbcd1db17b7af924af8c819"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.041097 25175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "bb378b093ad44c979509ee2e8f66f20f"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "42631e19f36f945a5af50534efb7234a"
server_key_iv: "10ae37962fbcd1db17b7af924af8c819"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.042465 25175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:54.043248 25189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.043430 25175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:54.043537 25175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "bb378b093ad44c979509ee2e8f66f20f"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "42631e19f36f945a5af50534efb7234a"
server_key_iv: "10ae37962fbcd1db17b7af924af8c819"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.043622 25175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:54.053805 25175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:54.054096 25175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:54.054243 25175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:54.054481 25175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:54.054809 25175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:54.054867 25175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.054909 25175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:54.054937 25175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.061112 25175 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:33925
I20260808 21:09:54.061134 25311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:33925 every 8 connection(s)
I20260808 21:09:54.061539 25175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:54.066571 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.062342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:40815 (local address 127.4.177.126:42859)
0808 21:09:54.062425 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.062427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.062646 (+ 219us) server_negotiation.cc:407] Connection header received
0808 21:09:54.063113 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.063115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.063124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.063197 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:54.063847 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.064344 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.065281 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.065459 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.065508 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.066359 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:54.066360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:54.066361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:54.066365 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:54.066376 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:54.066379 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:54.066416 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.066488 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.066508 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260808 21:09:54.066764 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.062320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42859 (local address 127.4.177.66:40815)
0808 21:09:54.062528 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.062561 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.063031 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.063265 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.063268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.063277 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:54.063776 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.063780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.064420 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.064421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.065202 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.065204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.065529 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.065530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.065637 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.066233 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:54.066233 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:54.066263 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:54.066285 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:54.066427 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:54.066429 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:54.066432 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.066479 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.066558 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260808 21:09:54.067240 25312 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42859
I20260808 21:09:54.067332 25312 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:54.067476 25312 heartbeater.cc:511] Master 127.4.177.126:42859 requested a full tablet report, sending...
I20260808 21:09:54.067867 24977 ts_manager.cc:194] Registered new tserver with Master: bb378b093ad44c979509ee2e8f66f20f (127.4.177.66:33925)
I20260808 21:09:54.068218 24977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.66:40815
I20260808 21:09:54.070410 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25175
I20260808 21:09:54.070502 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:54.070670 4805 external_mini_cluster.cc:1428] Setting key 68493433d945be7070df2f1ec59d0960
I20260808 21:09:54.071614 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--builtin_ntp_servers=127.4.177.84:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:54.154613 25315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:54.154855 25315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:54.154893 25315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:54.156095 25315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:54.156157 25315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:54.156201 25315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:54.157570 25315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:42859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:54.157850 25315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:54.158161 25315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:54.163177 25320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:54.163249 25324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:54.163393 25322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:54.163391 25315 server_base.cc:1034] running on GCE node
I20260808 21:09:54.163652 25315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:54.163892 25315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:54.165067 25315 hybrid_clock.cc:648] HybridClock initialized: now 1786223394165047 us; error 35 us; skew 500 ppm
I20260808 21:09:54.166328 25315 webserver.cc:460] Webserver started at http://127.4.177.67:35429/ using document root <none> and password file <none>
I20260808 21:09:54.166605 25315 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:54.166676 25315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:54.166823 25315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:54.167954 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "09af6feb3b1443638506856c8780ec41"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "2ff5a92f1053040d31fb70b7a00009af"
server_key_iv: "58d2f2eb41091e9b8cd16773f72575fd"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.168329 25315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "09af6feb3b1443638506856c8780ec41"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "2ff5a92f1053040d31fb70b7a00009af"
server_key_iv: "58d2f2eb41091e9b8cd16773f72575fd"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.169653 25315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:09:54.170387 25330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.170567 25315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:54.170670 25315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "09af6feb3b1443638506856c8780ec41"
format_stamp: "Formatted at 2026-08-08 21:09:54 on dist-test-slave-chsk"
server_key: "2ff5a92f1053040d31fb70b7a00009af"
server_key_iv: "58d2f2eb41091e9b8cd16773f72575fd"
server_key_version: "encryptionkey@0"
I20260808 21:09:54.170763 25315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:54.191314 25315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:54.191653 25315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:54.191825 25315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:54.192073 25315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:54.192425 25315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:54.192481 25315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.192515 25315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:54.192544 25315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:54.198318 25315 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:32889
I20260808 21:09:54.198360 25453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:32889 every 8 connection(s)
I20260808 21:09:54.198685 25315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:54.203989 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.199444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:47513 (local address 127.4.177.126:42859)
0808 21:09:54.199492 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.199494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.199766 (+ 272us) server_negotiation.cc:407] Connection header received
0808 21:09:54.200300 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.200302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.200315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.200374 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:54.201070 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.201578 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.202578 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.202773 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.202849 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.203714 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:54.203715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:54.203716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:54.203721 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:54.203736 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:54.203745 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:54.203803 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.203882 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.203908 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:09:54.204241 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.199441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:42859 (local address 127.4.177.67:47513)
0808 21:09:54.199642 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.199676 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.200191 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.200445 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.200449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.200459 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:54.200965 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.200970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.201679 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.201681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.202478 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.202480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.202856 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.202857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.202992 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.203557 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:54.203558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:54.203586 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:54.203608 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:54.203805 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:54.203811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:54.203815 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.203885 (+ 70us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.203983 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260808 21:09:54.204739 25455 heartbeater.cc:347] Connected to a master server at 127.4.177.126:42859
I20260808 21:09:54.204815 25455 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:54.205006 25455 heartbeater.cc:511] Master 127.4.177.126:42859 requested a full tablet report, sending...
I20260808 21:09:54.205394 24977 ts_manager.cc:194] Registered new tserver with Master: 09af6feb3b1443638506856c8780ec41 (127.4.177.67:32889)
I20260808 21:09:54.205756 24977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.67:47513
I20260808 21:09:54.207320 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25315
I20260808 21:09:54.207435 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:54.207649 4805 external_mini_cluster.cc:1428] Setting key 05df83053a792e271bd15a9d8a2a2385
I20260808 21:09:54.208590 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:54.212939 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.209916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.4.177.126:42859)
0808 21:09:54.209966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.209968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.210097 (+ 129us) server_negotiation.cc:407] Connection header received
0808 21:09:54.210171 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.210172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.210182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.210247 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:54.210427 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.210870 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.211700 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.211882 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.211952 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.212656 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:54.212657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:54.212658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:54.212662 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:54.212675 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:54.212680 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:54.212777 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.212844 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.212853 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 21:09:54.214393 24977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:54.215955 24977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:54.222373 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.4.177.67:32889)
0808 21:09:54.218792 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.218797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.218806 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:54.218839 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.218840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.218854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.218929 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:54.219099 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.220149 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.221456 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.221936 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222060 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222150 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.222251 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.222289 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":96,"threads_started":1}
I20260808 21:09:54.222361 25465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:32889 (local address 127.0.0.1:52908)
0808 21:09:54.218405 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.218415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.218459 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.218985 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.218986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.218988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:54.219029 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.219030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.220218 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.220219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.221349 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.221351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.222065 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.222148 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222163 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.222238 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.222290 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":3,"thread_start_us":109,"threads_started":1}
I20260808 21:09:54.222512 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:33925 (local address 127.0.0.1:49144)
0808 21:09:54.218139 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.218177 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.218236 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.218536 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.218539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.218540 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:54.218623 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.218625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.219992 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.219993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.221348 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.221351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.222231 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.222310 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222315 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.222357 (+ 42us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.222409 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260808 21:09:54.222723 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:39327 (local address 127.0.0.1:44912)
0808 21:09:54.218599 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.218610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.218658 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.219124 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.219125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.219127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:54.219188 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.219190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.220158 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.220159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.221259 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.221262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.222323 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.222380 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222382 (+ 2us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.222632 (+ 250us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.222669 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260808 21:09:54.222970 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.4.177.65:39327)
0808 21:09:54.218937 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.218942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.218952 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:54.218992 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.218994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.219009 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.219123 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:54.219247 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.220068 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.221352 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.222125 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222201 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222290 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.222835 (+ 545us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.222884 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260808 21:09:54.223101 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.218115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.4.177.66:33925)
0808 21:09:54.218289 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.218294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.218304 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:54.218345 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.218347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.218365 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.218455 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:54.218704 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.219882 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.221424 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.222124 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.222860 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.222955 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.222981 (+ 26us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.223023 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260808 21:09:54.223086 25382 tablet_service.cc:1467] Processing CreateTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d (DEFAULT_TABLE table=test-table [id=ce44885a2ef443f6b1e1394c2d7db71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:54.223498 25382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:54.223728 25239 tablet_service.cc:1467] Processing CreateTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d (DEFAULT_TABLE table=test-table [id=ce44885a2ef443f6b1e1394c2d7db71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:54.224033 25239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:54.226272 25469 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Bootstrap starting.
I20260808 21:09:54.226555 25470 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: Bootstrap starting.
I20260808 21:09:54.226967 25469 tablet_bootstrap.cc:654] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:54.226874 25105 tablet_service.cc:1467] Processing CreateTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d (DEFAULT_TABLE table=test-table [id=ce44885a2ef443f6b1e1394c2d7db71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:54.227229 25105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:54.227252 25470 tablet_bootstrap.cc:654] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:54.227371 25469 log.cc:824] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:54.227615 25470 log.cc:824] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:54.228153 25469 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: No bootstrap required, opened a new log
I20260808 21:09:54.228233 25469 ts_tablet_manager.cc:1397] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260808 21:09:54.230720 25473 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: Bootstrap starting.
I20260808 21:09:54.230854 25469 raft_consensus.cc:348] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.231053 25469 raft_consensus.cc:374] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:54.231101 25469 raft_consensus.cc:729] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09af6feb3b1443638506856c8780ec41, State: Initialized, Role: FOLLOWER
I20260808 21:09:54.231256 25469 consensus_queue.cc:260] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.231423 25473 tablet_bootstrap.cc:654] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:54.231527 25470 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: No bootstrap required, opened a new log
I20260808 21:09:54.231526 25469 ts_tablet_manager.cc:1428] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260808 21:09:54.231614 25470 ts_tablet_manager.cc:1397] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260808 21:09:54.231676 25455 heartbeater.cc:503] Master 127.4.177.126:42859 was elected leader, sending a full tablet report...
I20260808 21:09:54.231768 25473 log.cc:824] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:54.232774 25473 tablet_bootstrap.cc:492] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: No bootstrap required, opened a new log
I20260808 21:09:54.232847 25473 ts_tablet_manager.cc:1397] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:54.234017 25470 raft_consensus.cc:348] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.234207 25470 raft_consensus.cc:374] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:54.234269 25470 raft_consensus.cc:729] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb378b093ad44c979509ee2e8f66f20f, State: Initialized, Role: FOLLOWER
I20260808 21:09:54.234391 25470 consensus_queue.cc:260] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.234700 25470 ts_tablet_manager.cc:1428] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 21:09:54.234752 25312 heartbeater.cc:503] Master 127.4.177.126:42859 was elected leader, sending a full tablet report...
I20260808 21:09:54.235119 25473 raft_consensus.cc:348] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.235311 25473 raft_consensus.cc:374] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:54.235360 25473 raft_consensus.cc:729] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c6aae214e54a10a67177f6c482160f, State: Initialized, Role: FOLLOWER
I20260808 21:09:54.235476 25473 consensus_queue.cc:260] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.235756 25473 ts_tablet_manager.cc:1428] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 21:09:54.235873 25172 heartbeater.cc:503] Master 127.4.177.126:42859 was elected leader, sending a full tablet report...
I20260808 21:09:54.269897 25476 raft_consensus.cc:482] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:54.270038 25476 raft_consensus.cc:504] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.270473 25476 leader_election.cc:288] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09af6feb3b1443638506856c8780ec41 (127.4.177.67:32889), 87c6aae214e54a10a67177f6c482160f (127.4.177.65:39327)
I20260808 21:09:54.274343 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.270751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:55603 (local address 127.4.177.67:32889)
0808 21:09:54.270797 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.270799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.270812 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:09:54.270861 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.270862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.270869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.270940 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:54.271106 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.271922 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.273403 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.273949 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.274033 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.274042 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.274213 (+ 171us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.274262 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:09:54.274338 25314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.270666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:32889 (local address 127.4.177.66:55603)
0808 21:09:54.270724 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.270737 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.270786 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.270961 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.270962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.270964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:54.271040 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.271042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.272030 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.272032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.273304 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.273307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.274044 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.274045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.274124 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.274128 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.274195 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.274245 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":16}
I20260808 21:09:54.274333 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.270958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:57175 (local address 127.4.177.65:39327)
0808 21:09:54.271014 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.271016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.271024 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:54.271063 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.271064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.271073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.271136 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:54.271353 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.272058 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.273411 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.273978 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.274039 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.274045 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.274223 (+ 178us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.274264 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260808 21:09:54.274345 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.270795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:39327 (local address 127.4.177.66:57175)
0808 21:09:54.270962 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.270972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:54.271016 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:54.271165 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:54.271166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:54.271168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:54.271251 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.271253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.272144 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.272145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.273337 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:54.273339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.274077 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.274078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:54.274150 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.274153 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:54.274215 (+ 62us) client_negotiation.cc:240] Negotiation successful
0808 21:09:54.274245 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260808 21:09:54.274768 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a65b709ed4b4e23a33b95dbf1fcb90d" candidate_uuid: "bb378b093ad44c979509ee2e8f66f20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c6aae214e54a10a67177f6c482160f" is_pre_election: true
I20260808 21:09:54.274785 25402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a65b709ed4b4e23a33b95dbf1fcb90d" candidate_uuid: "bb378b093ad44c979509ee2e8f66f20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af6feb3b1443638506856c8780ec41" is_pre_election: true
I20260808 21:09:54.274946 25125 raft_consensus.cc:2454] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb378b093ad44c979509ee2e8f66f20f in term 0.
I20260808 21:09:54.274964 25402 raft_consensus.cc:2454] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb378b093ad44c979509ee2e8f66f20f in term 0.
I20260808 21:09:54.275204 25193 leader_election.cc:302] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c6aae214e54a10a67177f6c482160f, bb378b093ad44c979509ee2e8f66f20f; no voters:
I20260808 21:09:54.275352 25476 raft_consensus.cc:2781] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:54.275429 25476 raft_consensus.cc:482] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:54.275467 25476 raft_consensus.cc:3037] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:54.276283 25476 raft_consensus.cc:504] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.276432 25476 leader_election.cc:288] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [CANDIDATE]: Term 1 election: Requested vote from peers 09af6feb3b1443638506856c8780ec41 (127.4.177.67:32889), 87c6aae214e54a10a67177f6c482160f (127.4.177.65:39327)
I20260808 21:09:54.276563 25402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a65b709ed4b4e23a33b95dbf1fcb90d" candidate_uuid: "bb378b093ad44c979509ee2e8f66f20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09af6feb3b1443638506856c8780ec41"
I20260808 21:09:54.276628 25125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a65b709ed4b4e23a33b95dbf1fcb90d" candidate_uuid: "bb378b093ad44c979509ee2e8f66f20f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c6aae214e54a10a67177f6c482160f"
I20260808 21:09:54.276641 25402 raft_consensus.cc:3037] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:54.276710 25125 raft_consensus.cc:3037] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:54.277408 25125 raft_consensus.cc:2454] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb378b093ad44c979509ee2e8f66f20f in term 1.
I20260808 21:09:54.277419 25402 raft_consensus.cc:2454] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb378b093ad44c979509ee2e8f66f20f in term 1.
I20260808 21:09:54.277575 25193 leader_election.cc:302] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c6aae214e54a10a67177f6c482160f, bb378b093ad44c979509ee2e8f66f20f; no voters:
I20260808 21:09:54.277681 25476 raft_consensus.cc:2781] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:54.277844 25476 raft_consensus.cc:686] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 1 LEADER]: Becoming Leader. State: Replica: bb378b093ad44c979509ee2e8f66f20f, State: Running, Role: LEADER
I20260808 21:09:54.277949 25476 consensus_queue.cc:237] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } }
I20260808 21:09:54.278821 24977 catalog_manager.cc:5495] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f reported cstate change: term changed from 0 to 1, leader changed from <none> to bb378b093ad44c979509ee2e8f66f20f (127.4.177.66). New cstate: current_term: 1 leader_uuid: "bb378b093ad44c979509ee2e8f66f20f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb378b093ad44c979509ee2e8f66f20f" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 33925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 } health_report { overall_health: UNKNOWN } } }
I20260808 21:09:54.295522 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.293135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.4.177.66:33925)
0808 21:09:54.293198 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:54.293200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:54.293209 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:54.293262 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:54.293263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:54.293283 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:54.293345 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:54.293461 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.293921 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.294728 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:54.294911 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:54.294962 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:54.295090 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:54.295245 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:54.295330 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:54.295379 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 21:09:54.295387 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:09:54.297408 25402 raft_consensus.cc:1261] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 1 FOLLOWER]: Refusing update from remote peer bb378b093ad44c979509ee2e8f66f20f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:54.297408 25125 raft_consensus.cc:1261] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 1 FOLLOWER]: Refusing update from remote peer bb378b093ad44c979509ee2e8f66f20f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:54.297827 25476 consensus_queue.cc:1035] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09af6feb3b1443638506856c8780ec41" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:54.298014 25479 consensus_queue.cc:1035] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c6aae214e54a10a67177f6c482160f" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 39327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:54.300729 25483 mvcc.cc:205] Tried to move back new op lower bound from 7316371023039303680 to 7316371022963224576. Current Snapshot: MvccSnapshot[applied={T|T < 7316371023039303680}]
I20260808 21:09:54.301127 25485 mvcc.cc:205] Tried to move back new op lower bound from 7316371023039303680 to 7316371022963224576. Current Snapshot: MvccSnapshot[applied={T|T < 7316371023039303680}]
I20260808 21:09:54.303509 24975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:54.303601 24975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:54.304409 24975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7bf20de340d48a4a215f81b77f7ddc3: Sending DeleteTablet for 3 replicas of tablet 5a65b709ed4b4e23a33b95dbf1fcb90d
I20260808 21:09:54.304752 25239 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:54 UTC) from {username='slave'} at 127.0.0.1:49144
I20260808 21:09:54.304787 25382 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:54 UTC) from {username='slave'} at 127.0.0.1:52908
I20260808 21:09:54.304797 25105 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:54 UTC) from {username='slave'} at 127.0.0.1:44912
I20260808 21:09:54.305051 25495 tablet_replica.cc:330] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f: stopping tablet replica
I20260808 21:09:54.305167 25495 raft_consensus.cc:2229] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:54.305281 25495 raft_consensus.cc:2258] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 87c6aae214e54a10a67177f6c482160f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:54.305460 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25038
W20260808 21:09:54.318274 24963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:39327 (error 108)
I20260808 21:09:54.318447 25494 tablet_replica.cc:330] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: stopping tablet replica
I20260808 21:09:54.318612 25494 raft_consensus.cc:2229] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:54.318650 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25175
I20260808 21:09:54.318751 25494 raft_consensus.cc:2258] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:54.318945 25493 tablet_replica.cc:330] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f: stopping tablet replica
I20260808 21:09:54.319053 25493 raft_consensus.cc:2229] T 5a65b709ed4b4e23a33b95dbf1fcb90d P bb378b093ad44c979509ee2e8f66f20f [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:54.321224 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.321023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:39327 (local address 127.0.0.1:44928)
0808 21:09:54.321105 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.321173 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 21:09:54.321393 24963 catalog_manager.cc:4584] TS 87c6aae214e54a10a67177f6c482160f (127.4.177.65:39327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a65b709ed4b4e23a33b95dbf1fcb90d: Network error: Client connection negotiation failed: client connection to 127.4.177.65:39327: connect: Connection refused (error 111)
I20260808 21:09:54.321612 25494 ts_tablet_manager.cc:1905] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:54.323400 25494 ts_tablet_manager.cc:1918] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:54.323511 25494 log.cc:1196] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/5a65b709ed4b4e23a33b95dbf1fcb90d
I20260808 21:09:54.323778 25494 ts_tablet_manager.cc:1939] T 5a65b709ed4b4e23a33b95dbf1fcb90d P 09af6feb3b1443638506856c8780ec41: Deleting consensus metadata
I20260808 21:09:54.324283 24963 catalog_manager.cc:4841] TS 09af6feb3b1443638506856c8780ec41 (127.4.177.67:32889): tablet 5a65b709ed4b4e23a33b95dbf1fcb90d (table test-table [id=ce44885a2ef443f6b1e1394c2d7db71c]) successfully deleted
I20260808 21:09:54.325841 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25315
I20260808 21:09:54.325871 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:54.325735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:33925 (local address 127.0.0.1:49158)
0808 21:09:54.325801 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:54.325837 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.66:33925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260808 21:09:54.332638 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 24944
2026-08-08T21:09:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (714 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:54 dist-test-slave-chsk krb5kdc[25499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:54 dist-test-slave-chsk krb5kdc[25499](info): set up 2 sockets
Aug 08 21:09:54 dist-test-slave-chsk krb5kdc[25499](info): commencing operation
krb5kdc: starting...
W20260808 21:09:55.407853 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:09:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:09:55.528160 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33963
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:55.608223 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:55.608465 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:55.608487 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:55.609653 25512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:55.609695 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:55.609710 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:55.609722 25512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:55.609784 25512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:55.611249 25512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:33963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:55.611497 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:55.611726 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:55.616539 25519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:55.616636 25521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:55.616696 25512 server_base.cc:1034] running on GCE node
W20260808 21:09:55.616879 25518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:55.617148 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:55.617430 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:55.618589 25512 hybrid_clock.cc:648] HybridClock initialized: now 1786223395618589 us; error 39 us; skew 500 ppm
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:55.621088 25512 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:09:55.621558 25512 webserver.cc:460] Webserver started at http://127.4.177.126:33339/ using document root <none> and password file <none>
I20260808 21:09:55.621776 25512 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:55.621820 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:55.621978 25512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:55.623025 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "51e7911777344947a276045deca4e32b"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "05783fd34973c493c3a5303abf106643"
server_key_iv: "aed38fecfebf6474fe6f62d28c547533"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.623452 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "51e7911777344947a276045deca4e32b"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "05783fd34973c493c3a5303abf106643"
server_key_iv: "aed38fecfebf6474fe6f62d28c547533"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.624797 25512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:55.625522 25530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:55.625708 25512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:55.625785 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "51e7911777344947a276045deca4e32b"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "05783fd34973c493c3a5303abf106643"
server_key_iv: "aed38fecfebf6474fe6f62d28c547533"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.625869 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:55.646108 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:55.648028 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:55.648196 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:55.652594 25512 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:33963
I20260808 21:09:55.652635 25587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:33963 every 8 connection(s)
I20260808 21:09:55.653007 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:55.653877 25588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:55.656217 25588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: Bootstrap starting.
I20260808 21:09:55.656744 25588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:55.657013 25588 log.cc:824] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:55.657691 25588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: No bootstrap required, opened a new log
I20260808 21:09:55.659252 25588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } }
I20260808 21:09:55.659387 25588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:55.659409 25588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e7911777344947a276045deca4e32b, State: Initialized, Role: FOLLOWER
I20260808 21:09:55.659508 25588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } }
I20260808 21:09:55.659590 25588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:55.659626 25588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:55.659678 25588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:55.660230 25588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } }
I20260808 21:09:55.660354 25588 leader_election.cc:302] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e7911777344947a276045deca4e32b; no voters:
I20260808 21:09:55.660534 25588 leader_election.cc:288] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:55.660598 25591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:55.660820 25588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:55.660833 25591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [term 1 LEADER]: Becoming Leader. State: Replica: 51e7911777344947a276045deca4e32b, State: Running, Role: LEADER
I20260808 21:09:55.661005 25591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } }
I20260808 21:09:55.661435 25592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e7911777344947a276045deca4e32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } } }
I20260808 21:09:55.661571 25594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e7911777344947a276045deca4e32b. Latest consensus state: current_term: 1 leader_uuid: "51e7911777344947a276045deca4e32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e7911777344947a276045deca4e32b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 33963 } } }
I20260808 21:09:55.661608 25592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:55.661999 25594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:55.662299 25607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:55.662828 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25512
I20260808 21:09:55.662945 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:55.662945 25607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:55.663204 4805 external_mini_cluster.cc:1428] Setting key 2f5215f96359eeb9e98f1a10953a4c69
I20260808 21:09:55.665220 25607 catalog_manager.cc:1347] Generated new cluster ID: e3c8fc5d1b614e23b2fedf3152cf6434
I20260808 21:09:55.665282 25607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:55.675311 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:55.663697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.4.177.126:33963)
0808 21:09:55.664161 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:55.664170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:55.664201 (+ 31us) server_negotiation.cc:407] Connection header received
0808 21:09:55.664321 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:55.664328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:55.664392 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:55.664529 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:55.665401 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.666776 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.667936 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.668533 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.668643 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:55.671586 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:55.671591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:55.671591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:55.671603 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:55.673610 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:55.674006 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:55.674010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:55.674010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:55.674126 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:55.674330 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:55.674331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:55.674332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:55.674637 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:55.674829 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:55.674898 (+ 69us) server_negotiation.cc:299] Negotiation successful
0808 21:09:55.675038 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":167,"threads_started":1}
I20260808 21:09:55.685134 25607 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:55.685695 25607 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:55.688117 25607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: Generated new TSK 0
I20260808 21:09:55.688282 25607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.177.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
Entry for principal kudu/127.4.177.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab.
I20260808 21:09:55.718106 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:55.805629 25616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:55.805836 25616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:55.805861 25616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:55.807024 25616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:55.807065 25616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:55.807104 25616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:55.808662 25616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.65.keytab
--principal=kudu/127.4.177.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:55.808892 25616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:55.809129 25616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:55.814139 25622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:55.814148 25623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:55.814102 25625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:55.814493 25616 server_base.cc:1034] running on GCE node
I20260808 21:09:55.814684 25616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:55.814961 25616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:55.816144 25616 hybrid_clock.cc:648] HybridClock initialized: now 1786223395816106 us; error 64 us; skew 500 ppm
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:55.818614 25616 init.cc:377] Logged in from keytab as kudu/127.4.177.65@KRBTEST.COM (short username kudu)
I20260808 21:09:55.819049 25616 webserver.cc:460] Webserver started at http://127.4.177.65:40485/ using document root <none> and password file <none>
I20260808 21:09:55.819321 25616 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:55.819391 25616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:55.819530 25616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:55.820570 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "e39492633c45496e997a5ae358a6ee4e"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "d425bde97d441e62a7cbac3affbca588"
server_key_iv: "86b422035bf0d56bd4755b3d43b62ee8"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.820933 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "e39492633c45496e997a5ae358a6ee4e"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "d425bde97d441e62a7cbac3affbca588"
server_key_iv: "86b422035bf0d56bd4755b3d43b62ee8"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.822281 25616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:55.822999 25633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:55.823236 25616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:09:55.823359 25616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "e39492633c45496e997a5ae358a6ee4e"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "d425bde97d441e62a7cbac3affbca588"
server_key_iv: "86b422035bf0d56bd4755b3d43b62ee8"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.823433 25616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:55.841140 25616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:55.843235 25616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:55.843359 25616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:55.843562 25616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:55.843902 25616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:55.843937 25616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:55.843991 25616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:55.844017 25616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:55.849728 25616 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:37677
I20260808 21:09:55.849776 25755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:37677 every 8 connection(s)
I20260808 21:09:55.850119 25616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:55.853533 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25616
I20260808 21:09:55.853660 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:55.853885 4805 external_mini_cluster.cc:1428] Setting key fe0f97c3576e34488de18610d5968fa2
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.65@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:55.860978 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:55.850838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33963 (local address 127.4.177.65:45105)
0808 21:09:55.851024 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:55.851066 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:55.851581 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:55.851822 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:55.851825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:55.852393 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:55.852583 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:55.852587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.853250 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.853251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:55.854459 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:55.854462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.854858 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.854859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:55.854997 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:55.855583 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:55.855591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:55.857236 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:55.859441 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:55.859452 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:55.859454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:55.859719 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:55.859950 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:55.859951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:55.859951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:55.860060 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:55.860407 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:55.860413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:55.860581 (+ 168us) client_negotiation.cc:769] Sending connection context
0808 21:09:55.860669 (+ 88us) client_negotiation.cc:240] Negotiation successful
0808 21:09:55.860802 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260808 21:09:55.861014 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:55.850898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:45105 (local address 127.4.177.126:33963)
0808 21:09:55.850944 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:55.850946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:55.851161 (+ 215us) server_negotiation.cc:407] Connection header received
0808 21:09:55.851679 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:55.851680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:55.851694 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:55.851760 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:55.852670 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.853168 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.854545 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:55.854765 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:55.854846 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:55.857427 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:55.857429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:55.857430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:55.857440 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:55.859344 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:55.859807 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:55.859808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:55.859809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:55.859876 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:55.860129 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:55.860130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:55.860131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:55.860327 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:55.860404 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:55.860726 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 21:09:55.860840 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:09:55.861898 25756 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33963
I20260808 21:09:55.862036 25756 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:55.862223 25756 heartbeater.cc:511] Master 127.4.177.126:33963 requested a full tablet report, sending...
I20260808 21:09:55.862836 25548 ts_manager.cc:194] Registered new tserver with Master: e39492633c45496e997a5ae358a6ee4e (127.4.177.65:37677)
I20260808 21:09:55.863536 25548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.65@KRBTEST.COM'} at 127.4.177.65:45105
WARNING: no policy specified for kudu/127.4.177.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
Entry for principal kudu/127.4.177.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab.
I20260808 21:09:55.882493 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:55.965071 25761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:55.965338 25761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:55.965385 25761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:55.966593 25761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:55.966663 25761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:55.966711 25761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:55.968341 25761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.66.keytab
--principal=kudu/127.4.177.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:55.968621 25761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:55.968873 25761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:55.973975 25766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:55.973989 25767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:55.973975 25769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:55.974237 25761 server_base.cc:1034] running on GCE node
I20260808 21:09:55.974493 25761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:55.974823 25761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:55.975986 25761 hybrid_clock.cc:648] HybridClock initialized: now 1786223395975974 us; error 43 us; skew 500 ppm
Aug 08 21:09:55 dist-test-slave-chsk krb5kdc[25499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:55.978544 25761 init.cc:377] Logged in from keytab as kudu/127.4.177.66@KRBTEST.COM (short username kudu)
I20260808 21:09:55.978986 25761 webserver.cc:460] Webserver started at http://127.4.177.66:34737/ using document root <none> and password file <none>
I20260808 21:09:55.979264 25761 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:55.979333 25761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:55.979489 25761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:55.980566 25761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "1dac24a7338f4d8790970e1a5a50b2d5"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "c4397a7d95b6a46b99ee5f05cdcf6699"
server_key_iv: "c6e37b019de24a8fef1dd0633bd8376a"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.980942 25761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "1dac24a7338f4d8790970e1a5a50b2d5"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "c4397a7d95b6a46b99ee5f05cdcf6699"
server_key_iv: "c6e37b019de24a8fef1dd0633bd8376a"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.982270 25761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:55.982957 25777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:55.983124 25761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:55.983279 25761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "1dac24a7338f4d8790970e1a5a50b2d5"
format_stamp: "Formatted at 2026-08-08 21:09:55 on dist-test-slave-chsk"
server_key: "c4397a7d95b6a46b99ee5f05cdcf6699"
server_key_iv: "c6e37b019de24a8fef1dd0633bd8376a"
server_key_version: "encryptionkey@0"
I20260808 21:09:55.983363 25761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:56.000896 25761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:56.002898 25761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:56.003079 25761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:56.003394 25761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:56.003755 25761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:56.003811 25761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.003854 25761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:56.003880 25761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.009809 25761 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:45695
I20260808 21:09:56.009892 25896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:45695 every 8 connection(s)
I20260808 21:09:56.010226 25761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
Aug 08 21:09:56 dist-test-slave-chsk krb5kdc[25499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.66@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:56.018396 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25761
I20260808 21:09:56.018532 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:56.018787 4805 external_mini_cluster.cc:1428] Setting key ee135057bf9c8e41b3c4752fe7e54cb3
I20260808 21:09:56.020329 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.011052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:35915 (local address 127.4.177.126:33963)
0808 21:09:56.011118 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.011120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.011345 (+ 225us) server_negotiation.cc:407] Connection header received
0808 21:09:56.011888 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.011889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.011904 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.011958 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.012965 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.013580 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.014729 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.014938 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.015020 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.017264 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.017266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.017266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:56.017274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.018602 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.019208 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.019210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.019210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.019289 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.019604 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.019605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.019605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.019773 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.019832 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.020181 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.020263 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:09:56.020634 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.011040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33963 (local address 127.4.177.66:35915)
0808 21:09:56.011204 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.011254 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.011777 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.012037 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.012041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.012645 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:56.012852 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.012857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.013690 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.013691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.014626 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.014629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.015029 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.015030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.015225 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.015713 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:56.015721 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:56.017154 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:56.018684 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:56.018692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:56.018694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:56.019069 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:56.019361 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:56.019362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:56.019363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:56.019514 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:56.019848 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:56.019856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:56.020031 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.020144 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.020338 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":129,"threads_started":1}
I20260808 21:09:56.021102 25897 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33963
I20260808 21:09:56.021221 25897 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:56.021430 25897 heartbeater.cc:511] Master 127.4.177.126:33963 requested a full tablet report, sending...
I20260808 21:09:56.021835 25548 ts_manager.cc:194] Registered new tserver with Master: 1dac24a7338f4d8790970e1a5a50b2d5 (127.4.177.66:45695)
I20260808 21:09:56.022254 25548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.66@KRBTEST.COM'} at 127.4.177.66:35915
WARNING: no policy specified for kudu/127.4.177.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
Entry for principal kudu/127.4.177.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab.
I20260808 21:09:56.046157 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:56.131099 25903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:56.131398 25903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:56.131422 25903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:56.132532 25903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:56.132568 25903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:56.132601 25903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:56.133970 25903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.67.keytab
--principal=kudu/127.4.177.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:33963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:56.134153 25903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:56.134330 25903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:56.139503 25908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.139560 25903 server_base.cc:1034] running on GCE node
W20260808 21:09:56.139521 25909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:56.139541 25912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.139881 25903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:56.140190 25903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:56.141350 25903 hybrid_clock.cc:648] HybridClock initialized: now 1786223396141324 us; error 36 us; skew 500 ppm
Aug 08 21:09:56 dist-test-slave-chsk krb5kdc[25499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:56.143909 25903 init.cc:377] Logged in from keytab as kudu/127.4.177.67@KRBTEST.COM (short username kudu)
I20260808 21:09:56.144397 25903 webserver.cc:460] Webserver started at http://127.4.177.67:45329/ using document root <none> and password file <none>
I20260808 21:09:56.144627 25903 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:56.144673 25903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:56.144845 25903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:56.145895 25903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "01fb52ca65254a039d793eb14ef1ab19"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "4cc9ee9dded5a85a47bb81fef0d7f602"
server_key_iv: "a896eca0ffd6f217f4debdf03ad474ee"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.146262 25903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "01fb52ca65254a039d793eb14ef1ab19"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "4cc9ee9dded5a85a47bb81fef0d7f602"
server_key_iv: "a896eca0ffd6f217f4debdf03ad474ee"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.147681 25903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:56.148378 25920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.148571 25903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.148661 25903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "01fb52ca65254a039d793eb14ef1ab19"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "4cc9ee9dded5a85a47bb81fef0d7f602"
server_key_iv: "a896eca0ffd6f217f4debdf03ad474ee"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.148715 25903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:56.169632 25903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:56.172035 25903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:56.172149 25903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:56.172338 25903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:56.172662 25903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:56.172698 25903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.172739 25903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:56.172775 25903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.178725 25903 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:46211
I20260808 21:09:56.178826 26040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:46211 every 8 connection(s)
I20260808 21:09:56.179126 25903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:56.182005 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 25903
I20260808 21:09:56.182149 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:56.182387 4805 external_mini_cluster.cc:1428] Setting key 66e3c4b7f4ff82706d91abd4dafddc28
Aug 08 21:09:56 dist-test-slave-chsk krb5kdc[25499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.67@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:09:56.188506 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.179924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:55445 (local address 127.4.177.126:33963)
0808 21:09:56.179993 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.179995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.180218 (+ 223us) server_negotiation.cc:407] Connection header received
0808 21:09:56.180678 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.180679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.180690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.180744 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.181670 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.182216 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.183619 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.183812 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.183885 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.185894 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.185895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.185896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:56.185903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.187305 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.187660 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.187661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.187661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.187711 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.187900 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.187901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.187901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.188003 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.188055 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.188377 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.188441 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 21:09:56.188760 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.179892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:33963 (local address 127.4.177.67:55445)
0808 21:09:56.180079 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.180122 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.180597 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.180805 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.180808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.181400 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:56.181593 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.181597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.182308 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.182309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.183546 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.183549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.183909 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.183910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.184007 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.184444 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:09:56.184452 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:56.185791 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:56.187373 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:56.187379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:56.187380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:56.187593 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:56.187752 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:09:56.187753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:09:56.187753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:09:56.187837 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:09:56.188068 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:56.188075 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:56.188252 (+ 177us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.188344 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.188465 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260808 21:09:56.189245 26041 heartbeater.cc:347] Connected to a master server at 127.4.177.126:33963
I20260808 21:09:56.189402 26041 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:56.189584 26041 heartbeater.cc:511] Master 127.4.177.126:33963 requested a full tablet report, sending...
I20260808 21:09:56.189927 25548 ts_manager.cc:194] Registered new tserver with Master: 01fb52ca65254a039d793eb14ef1ab19 (127.4.177.67:46211)
I20260808 21:09:56.190306 25548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.177.67@KRBTEST.COM'} at 127.4.177.67:55445
I20260808 21:09:56.194101 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:56.200872 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.195513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.4.177.126:33963)
0808 21:09:56.195553 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.195554 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.195623 (+ 69us) server_negotiation.cc:407] Connection header received
0808 21:09:56.195673 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.195674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.195681 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.195726 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.196211 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.196644 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.197314 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.197486 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.197534 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.198713 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.198714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.198714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:56.198719 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.199920 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.200218 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.200219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.200219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.200273 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:56.200433 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:56.200434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:56.200434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:56.200567 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.200641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.200753 (+ 112us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.200810 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:09:56.202486 25548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:56.204223 25548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:56.210669 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.4.177.66:45695)
0808 21:09:56.207034 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.207039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.207051 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:09:56.207096 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.207099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.207115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.207248 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:56.207565 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208266 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.209622 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.210259 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.210348 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.210411 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.210547 (+ 136us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.210589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":87,"threads_started":1}
I20260808 21:09:56.210660 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:45695 (local address 127.0.0.1:55736)
0808 21:09:56.206912 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.206921 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.206961 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.207374 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.207377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.207379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:56.207458 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.207460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208371 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.208372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.209524 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.209527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.210346 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.210347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.210425 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.210439 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.210531 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.210589 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":6,"thread_start_us":113,"threads_started":1}
I20260808 21:09:56.211097 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:46211 (local address 127.0.0.1:39780)
0808 21:09:56.207035 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.207041 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.207106 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.207420 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.207421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.207423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:56.207488 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.207489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208786 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.208787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.209825 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.209828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.210847 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.210848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.210906 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.210910 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.210966 (+ 56us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.211005 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":32,"thread_start_us":139,"threads_started":1}
I20260808 21:09:56.211123 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.4.177.67:46211)
0808 21:09:56.207034 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.207039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.207113 (+ 74us) server_negotiation.cc:407] Connection header received
0808 21:09:56.207212 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.207215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.207233 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.207300 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:56.207572 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208651 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.209919 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.210668 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.210741 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.210803 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.210997 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.211049 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260808 21:09:56.211742 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37677 (local address 127.0.0.1:50334)
0808 21:09:56.206780 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.206822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.206890 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.207379 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.207379 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.207381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:56.207459 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.207460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208784 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.208785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.209842 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.209844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.211525 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.211526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.211591 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.211595 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.211643 (+ 48us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.211679 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":34,"thread_start_us":119,"threads_started":1}
I20260808 21:09:56.211642 25827 tablet_service.cc:1467] Processing CreateTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc (DEFAULT_TABLE table=test-table [id=58057af2f9964afc9b00d113e4cd290d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:56.211907 25972 tablet_service.cc:1467] Processing CreateTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc (DEFAULT_TABLE table=test-table [id=58057af2f9964afc9b00d113e4cd290d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:56.212052 25827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d7a49829ca4706968a182cd4f6e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:56.212108 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.206824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.4.177.65:37677)
0808 21:09:56.207128 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.207132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.207167 (+ 35us) server_negotiation.cc:407] Connection header received
0808 21:09:56.207207 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.207209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.207225 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.207296 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:56.207532 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.208676 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.209951 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.210667 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.211091 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.211954 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.211982 (+ 28us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.212028 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":77,"threads_started":1}
I20260808 21:09:56.212270 25972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d7a49829ca4706968a182cd4f6e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:56.214862 26060 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Bootstrap starting.
I20260808 21:09:56.215057 25683 tablet_service.cc:1467] Processing CreateTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc (DEFAULT_TABLE table=test-table [id=58057af2f9964afc9b00d113e4cd290d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:56.215207 26061 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Bootstrap starting.
I20260808 21:09:56.215445 25683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d7a49829ca4706968a182cd4f6e8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:56.215637 26060 tablet_bootstrap.cc:654] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:56.215839 26061 tablet_bootstrap.cc:654] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:56.216178 26060 log.cc:824] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:56.216420 26061 log.cc:824] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:56.217000 26060 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: No bootstrap required, opened a new log
I20260808 21:09:56.217092 26060 ts_tablet_manager.cc:1397] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:56.218684 26064 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Bootstrap starting.
I20260808 21:09:56.218822 26060 raft_consensus.cc:348] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.219014 26060 raft_consensus.cc:374] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:56.219064 26060 raft_consensus.cc:729] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fb52ca65254a039d793eb14ef1ab19, State: Initialized, Role: FOLLOWER
I20260808 21:09:56.219203 26060 consensus_queue.cc:260] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.219316 26064 tablet_bootstrap.cc:654] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:56.219517 26060 ts_tablet_manager.cc:1428] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:09:56.219704 26041 heartbeater.cc:503] Master 127.4.177.126:33963 was elected leader, sending a full tablet report...
I20260808 21:09:56.219725 26061 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: No bootstrap required, opened a new log
I20260808 21:09:56.219790 26061 ts_tablet_manager.cc:1397] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260808 21:09:56.221341 26061 raft_consensus.cc:348] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.221478 26061 raft_consensus.cc:374] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:56.221500 26061 raft_consensus.cc:729] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dac24a7338f4d8790970e1a5a50b2d5, State: Initialized, Role: FOLLOWER
I20260808 21:09:56.221612 26061 consensus_queue.cc:260] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.221839 26061 ts_tablet_manager.cc:1428] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:56.221904 25897 heartbeater.cc:503] Master 127.4.177.126:33963 was elected leader, sending a full tablet report...
I20260808 21:09:56.222656 26064 log.cc:824] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:56.223459 26064 tablet_bootstrap.cc:492] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: No bootstrap required, opened a new log
I20260808 21:09:56.223515 26064 ts_tablet_manager.cc:1397] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260808 21:09:56.224805 26064 raft_consensus.cc:348] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.224941 26064 raft_consensus.cc:374] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:56.224963 26064 raft_consensus.cc:729] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39492633c45496e997a5ae358a6ee4e, State: Initialized, Role: FOLLOWER
I20260808 21:09:56.225072 26064 consensus_queue.cc:260] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.225310 26064 ts_tablet_manager.cc:1428] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260808 21:09:56.225374 25756 heartbeater.cc:503] Master 127.4.177.126:33963 was elected leader, sending a full tablet report...
W20260808 21:09:56.260810 25898 tablet.cc:2369] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 21:09:56.350950 25757 tablet.cc:2369] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:56.404052 26065 raft_consensus.cc:482] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:56.404186 26065 raft_consensus.cc:504] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.404661 26065 leader_election.cc:288] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e39492633c45496e997a5ae358a6ee4e (127.4.177.65:37677), 1dac24a7338f4d8790970e1a5a50b2d5 (127.4.177.66:45695)
I20260808 21:09:56.407823 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.404848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37677 (local address 127.4.177.67:37633)
0808 21:09:56.404902 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.404915 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.404962 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.405128 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.405130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.405132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:56.405205 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.405207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.406044 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.406045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.406859 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.406861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.407546 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.407547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.407637 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.407640 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.407687 (+ 47us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.407740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":4}
I20260808 21:09:56.407833 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.404929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:37633 (local address 127.4.177.65:37677)
0808 21:09:56.404979 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.404981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.404992 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:09:56.405024 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.405025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.405036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.405094 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:56.405321 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.405933 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.406959 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.407450 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.407505 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.407511 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.407718 (+ 207us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.407761 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:09:56.407871 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.405119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:59883 (local address 127.4.177.66:45695)
0808 21:09:56.405171 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.405173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.405184 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:09:56.405263 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.405264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.405292 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.405443 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:56.405604 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.406147 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.407094 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.407508 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.407579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.407588 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.407760 (+ 172us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.407797 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260808 21:09:56.407902 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.404940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:45695 (local address 127.4.177.67:59883)
0808 21:09:56.405135 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.405144 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.405196 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.405453 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.405453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.405455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:56.405519 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.405521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.406233 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.406234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.406996 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.406998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.407596 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.407597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.407660 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.407663 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.407739 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.407773 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":128,"threads_started":1}
I20260808 21:09:56.408203 25706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d7a49829ca4706968a182cd4f6e8cc" candidate_uuid: "01fb52ca65254a039d793eb14ef1ab19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39492633c45496e997a5ae358a6ee4e" is_pre_election: true
I20260808 21:09:56.408273 25849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d7a49829ca4706968a182cd4f6e8cc" candidate_uuid: "01fb52ca65254a039d793eb14ef1ab19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" is_pre_election: true
I20260808 21:09:56.408380 25706 raft_consensus.cc:2454] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01fb52ca65254a039d793eb14ef1ab19 in term 0.
I20260808 21:09:56.408447 25849 raft_consensus.cc:2454] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01fb52ca65254a039d793eb14ef1ab19 in term 0.
I20260808 21:09:56.408615 25923 leader_election.cc:302] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fb52ca65254a039d793eb14ef1ab19, e39492633c45496e997a5ae358a6ee4e; no voters:
I20260808 21:09:56.408788 26065 raft_consensus.cc:2781] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:56.408843 26065 raft_consensus.cc:482] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:56.408869 26065 raft_consensus.cc:3037] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:56.409600 26065 raft_consensus.cc:504] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.409750 26065 leader_election.cc:288] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [CANDIDATE]: Term 1 election: Requested vote from peers e39492633c45496e997a5ae358a6ee4e (127.4.177.65:37677), 1dac24a7338f4d8790970e1a5a50b2d5 (127.4.177.66:45695)
I20260808 21:09:56.409940 25849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d7a49829ca4706968a182cd4f6e8cc" candidate_uuid: "01fb52ca65254a039d793eb14ef1ab19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dac24a7338f4d8790970e1a5a50b2d5"
I20260808 21:09:56.409996 25706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d7a49829ca4706968a182cd4f6e8cc" candidate_uuid: "01fb52ca65254a039d793eb14ef1ab19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39492633c45496e997a5ae358a6ee4e"
I20260808 21:09:56.410063 25849 raft_consensus.cc:3037] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:56.410122 25706 raft_consensus.cc:3037] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:56.410844 25849 raft_consensus.cc:2454] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01fb52ca65254a039d793eb14ef1ab19 in term 1.
I20260808 21:09:56.410877 25706 raft_consensus.cc:2454] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01fb52ca65254a039d793eb14ef1ab19 in term 1.
I20260808 21:09:56.411072 25923 leader_election.cc:302] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01fb52ca65254a039d793eb14ef1ab19, e39492633c45496e997a5ae358a6ee4e; no voters:
I20260808 21:09:56.411170 26065 raft_consensus.cc:2781] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:56.411332 26065 raft_consensus.cc:686] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 1 LEADER]: Becoming Leader. State: Replica: 01fb52ca65254a039d793eb14ef1ab19, State: Running, Role: LEADER
I20260808 21:09:56.411466 26065 consensus_queue.cc:237] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } }
I20260808 21:09:56.412340 25546 catalog_manager.cc:5495] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01fb52ca65254a039d793eb14ef1ab19 (127.4.177.67). New cstate: current_term: 1 leader_uuid: "01fb52ca65254a039d793eb14ef1ab19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01fb52ca65254a039d793eb14ef1ab19" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 46211 } health_report { overall_health: HEALTHY } } }
W20260808 21:09:56.429685 26042 tablet.cc:2369] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 21:09:56.436574 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.434302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.4.177.67:46211)
0808 21:09:56.434366 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.434368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.434377 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:56.434427 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.434428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.434444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.434502 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:56.434610 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.435084 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.435850 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.436035 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.436094 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.436226 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:56.436327 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:56.436387 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.436416 (+ 29us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.436436 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260808 21:09:56.438325 25849 raft_consensus.cc:1261] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 1 FOLLOWER]: Refusing update from remote peer 01fb52ca65254a039d793eb14ef1ab19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:56.438364 25706 raft_consensus.cc:1261] T 07d7a49829ca4706968a182cd4f6e8cc P e39492633c45496e997a5ae358a6ee4e [term 1 FOLLOWER]: Refusing update from remote peer 01fb52ca65254a039d793eb14ef1ab19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:56.438714 26065 consensus_queue.cc:1035] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39492633c45496e997a5ae358a6ee4e" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 37677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:56.438863 26071 consensus_queue.cc:1035] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dac24a7338f4d8790970e1a5a50b2d5" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 45695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:56.441486 26075 mvcc.cc:205] Tried to move back new op lower bound from 7316371031808729088 to 7316371031702003712. Current Snapshot: MvccSnapshot[applied={T|T < 7316371031808729088}]
I20260808 21:09:56.441922 26076 mvcc.cc:205] Tried to move back new op lower bound from 7316371031808729088 to 7316371031702003712. Current Snapshot: MvccSnapshot[applied={T|T < 7316371031808729088 or (T in {7316371031808729088})}]
I20260808 21:09:56.444095 25548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:56.444195 25548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:56.444973 25548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51e7911777344947a276045deca4e32b: Sending DeleteTablet for 3 replicas of tablet 07d7a49829ca4706968a182cd4f6e8cc
I20260808 21:09:56.445319 25827 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:56 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:55736
I20260808 21:09:56.445319 25972 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:56 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:39780
I20260808 21:09:56.445580 25683 tablet_service.cc:1514] Processing DeleteTablet for tablet 07d7a49829ca4706968a182cd4f6e8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:56 UTC) from {username='kudu', principal='kudu/127.4.177.126@KRBTEST.COM'} at 127.0.0.1:50334
I20260808 21:09:56.445626 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25616
I20260808 21:09:56.446857 26083 tablet_replica.cc:330] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: stopping tablet replica
I20260808 21:09:56.446983 26083 raft_consensus.cc:2229] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:56.447105 26083 raft_consensus.cc:2258] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:56.447229 26084 tablet_replica.cc:330] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: stopping tablet replica
I20260808 21:09:56.447328 26084 raft_consensus.cc:2229] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:56.447427 26084 raft_consensus.cc:2258] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:56.447890 26084 ts_tablet_manager.cc:1905] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:56.449431 26084 ts_tablet_manager.cc:1918] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:56.449431 26083 ts_tablet_manager.cc:1905] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:56.449579 26084 log.cc:1196] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/07d7a49829ca4706968a182cd4f6e8cc
I20260808 21:09:56.450011 26084 ts_tablet_manager.cc:1939] T 07d7a49829ca4706968a182cd4f6e8cc P 1dac24a7338f4d8790970e1a5a50b2d5: Deleting consensus metadata
I20260808 21:09:56.450513 25531 catalog_manager.cc:4841] TS 1dac24a7338f4d8790970e1a5a50b2d5 (127.4.177.66:45695): tablet 07d7a49829ca4706968a182cd4f6e8cc (table test-table [id=58057af2f9964afc9b00d113e4cd290d]) successfully deleted
I20260808 21:09:56.451262 26083 ts_tablet_manager.cc:1918] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:56.451347 26083 log.cc:1196] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/07d7a49829ca4706968a182cd4f6e8cc
I20260808 21:09:56.451645 26083 ts_tablet_manager.cc:1939] T 07d7a49829ca4706968a182cd4f6e8cc P 01fb52ca65254a039d793eb14ef1ab19: Deleting consensus metadata
I20260808 21:09:56.452039 25534 catalog_manager.cc:4841] TS 01fb52ca65254a039d793eb14ef1ab19 (127.4.177.67:46211): tablet 07d7a49829ca4706968a182cd4f6e8cc (table test-table [id=58057af2f9964afc9b00d113e4cd290d]) successfully deleted
W20260808 21:09:56.455904 25533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:37677 (error 108)
I20260808 21:09:56.456120 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25761
I20260808 21:09:56.463850 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.463665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:37677 (local address 127.0.0.1:50336)
0808 21:09:56.463732 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.463802 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 21:09:56.464002 25533 catalog_manager.cc:4584] TS e39492633c45496e997a5ae358a6ee4e (127.4.177.65:37677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07d7a49829ca4706968a182cd4f6e8cc: Network error: Client connection negotiation failed: client connection to 127.4.177.65:37677: connect: Connection refused (error 111)
I20260808 21:09:56.465047 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25903
I20260808 21:09:56.470762 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 25512
2026-08-08T21:09:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2137 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T21:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:56Z Disabled control of system clock
I20260808 21:09:56.500586 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44271
--webserver_interface=127.4.177.126
--webserver_port=0
--builtin_ntp_servers=127.4.177.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:56.582729 26090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:56.582945 26090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:56.582966 26090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:56.584147 26090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:56.584200 26090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:56.584213 26090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:56.584226 26090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:56.584262 26090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:56.585657 26090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:44271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:56.585918 26090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:56.586241 26090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:56.591089 26096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:56.591115 26097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:56.591099 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.591370 26090 server_base.cc:1034] running on GCE node
I20260808 21:09:56.591588 26090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:56.591902 26090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:56.593070 26090 hybrid_clock.cc:648] HybridClock initialized: now 1786223396593067 us; error 38 us; skew 500 ppm
I20260808 21:09:56.594370 26090 webserver.cc:460] Webserver started at http://127.4.177.126:45843/ using document root <none> and password file <none>
I20260808 21:09:56.594653 26090 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:56.594722 26090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:56.594870 26090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:56.595968 26090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "49f665b0694846799170e8d1650a897e"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "c1c6e505b54ea451a0c0164cde73ed7f"
server_key_iv: "be911b4f5dadb86d35ab7dfccfc3e077"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.596325 26090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "49f665b0694846799170e8d1650a897e"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "c1c6e505b54ea451a0c0164cde73ed7f"
server_key_iv: "be911b4f5dadb86d35ab7dfccfc3e077"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.597621 26090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260808 21:09:56.598380 26105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.598588 26090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:09:56.598687 26090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "49f665b0694846799170e8d1650a897e"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "c1c6e505b54ea451a0c0164cde73ed7f"
server_key_iv: "be911b4f5dadb86d35ab7dfccfc3e077"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.598767 26090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:56.616134 26090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:56.616472 26090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:56.616622 26090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:56.620785 26090 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:44271
I20260808 21:09:56.620821 26161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:44271 every 8 connection(s)
I20260808 21:09:56.621147 26090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:56.622126 26163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:56.624353 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: Bootstrap starting.
I20260808 21:09:56.624884 26163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:56.625245 26163 log.cc:824] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:56.625262 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26090
I20260808 21:09:56.625357 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:56.625572 4805 external_mini_cluster.cc:1428] Setting key ebeccf2f9f648e7b8aea3c66f459c755
I20260808 21:09:56.626014 26163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: No bootstrap required, opened a new log
I20260808 21:09:56.628852 26163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } }
I20260808 21:09:56.629010 26163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:56.629040 26163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f665b0694846799170e8d1650a897e, State: Initialized, Role: FOLLOWER
I20260808 21:09:56.629144 26163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } }
I20260808 21:09:56.629259 26163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:56.629309 26163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:56.629386 26163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:56.629907 26163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } }
I20260808 21:09:56.630025 26163 leader_election.cc:302] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f665b0694846799170e8d1650a897e; no voters:
I20260808 21:09:56.630240 26163 leader_election.cc:288] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:56.630283 26168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:56.630402 26168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [term 1 LEADER]: Becoming Leader. State: Replica: 49f665b0694846799170e8d1650a897e, State: Running, Role: LEADER
I20260808 21:09:56.630496 26168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } }
I20260808 21:09:56.630522 26163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:56.630858 26168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f665b0694846799170e8d1650a897e. Latest consensus state: current_term: 1 leader_uuid: "49f665b0694846799170e8d1650a897e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } } }
I20260808 21:09:56.630928 26168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:56.631266 26169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f665b0694846799170e8d1650a897e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f665b0694846799170e8d1650a897e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44271 } } }
I20260808 21:09:56.631531 26175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:56.631438 26169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:56.632390 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.626085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.4.177.126:44271)
0808 21:09:56.626290 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.626298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.626335 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:09:56.627310 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.627319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.627392 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.627540 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.627752 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.629164 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.630129 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.630770 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.630893 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.631633 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.631637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.631637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:56.631848 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.631895 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:56.631911 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.631976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.632101 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.632181 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260808 21:09:56.632838 26175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:56.634512 26175 catalog_manager.cc:1347] Generated new cluster ID: adb6614af19e47ebb1f6188830243ec4
I20260808 21:09:56.634577 26175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:56.658258 26175 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:56.658787 26175 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:56.663558 26175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: Generated new TSK 0
I20260808 21:09:56.663745 26175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:09:56.671000 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.65:0
--local_ip_for_outbound_sockets=127.4.177.65
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--builtin_ntp_servers=127.4.177.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:56.753535 26190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:56.753753 26190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:56.753775 26190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:56.754853 26190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:56.754887 26190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:56.754930 26190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.65
I20260808 21:09:56.756219 26190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.177.65
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.65
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:56.756434 26190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:56.756675 26190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:56.761431 26195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.761478 26190 server_base.cc:1034] running on GCE node
W20260808 21:09:56.761475 26196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:56.761445 26198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.761916 26190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:56.762154 26190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:56.763293 26190 hybrid_clock.cc:648] HybridClock initialized: now 1786223396763274 us; error 36 us; skew 500 ppm
I20260808 21:09:56.764578 26190 webserver.cc:460] Webserver started at http://127.4.177.65:35627/ using document root <none> and password file <none>
I20260808 21:09:56.764829 26190 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:56.764895 26190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:56.765050 26190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:56.766084 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data/instance:
uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "8a015e222043b4388d202d1eaf997296"
server_key_iv: "84014267bb34a6a4ca2476b25fb53ffd"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.766470 26190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance:
uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "8a015e222043b4388d202d1eaf997296"
server_key_iv: "84014267bb34a6a4ca2476b25fb53ffd"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.767829 26190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:56.768537 26206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.768725 26190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:56.768808 26190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal
uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "8a015e222043b4388d202d1eaf997296"
server_key_iv: "84014267bb34a6a4ca2476b25fb53ffd"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.768877 26190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:56.781168 26190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:56.781452 26190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:56.781595 26190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:56.781805 26190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:56.782122 26190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:56.782155 26190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.782205 26190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:56.782240 26190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.788125 26190 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.65:40551
I20260808 21:09:56.788214 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.65:40551 every 8 connection(s)
I20260808 21:09:56.788540 26190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/data/info.pb
I20260808 21:09:56.793854 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.789292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:49687 (local address 127.4.177.126:44271)
0808 21:09:56.789339 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.789341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.789592 (+ 251us) server_negotiation.cc:407] Connection header received
0808 21:09:56.790140 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.790141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.790158 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.790237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.790941 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.791527 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.792499 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.792688 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.792796 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.793564 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.793565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.793566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:56.793571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.793585 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:56.793589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.793645 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.793746 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.793776 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:09:56.794078 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.789276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44271 (local address 127.4.177.65:49687)
0808 21:09:56.789467 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.789500 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.790008 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.790309 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.790314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.790323 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:56.790843 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.790847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.791626 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.791627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.792405 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.792407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.792805 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.792806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.792937 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.793424 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:56.793425 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:56.793452 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:56.793473 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:56.793649 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:56.793653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:56.793656 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.793734 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.793833 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260808 21:09:56.794591 26330 heartbeater.cc:347] Connected to a master server at 127.4.177.126:44271
I20260808 21:09:56.794692 26330 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:56.794832 26330 heartbeater.cc:511] Master 127.4.177.126:44271 requested a full tablet report, sending...
I20260808 21:09:56.795369 26124 ts_manager.cc:194] Registered new tserver with Master: cea5a6aed16444d0a0dddcdbaa3f7ca7 (127.4.177.65:40551)
I20260808 21:09:56.796006 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.65:49687
I20260808 21:09:56.796926 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26190
I20260808 21:09:56.797034 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-0/wal/instance
I20260808 21:09:56.797245 4805 external_mini_cluster.cc:1428] Setting key a02b74080a699e12a70a073485b358bc
I20260808 21:09:56.798179 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.66:0
--local_ip_for_outbound_sockets=127.4.177.66
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--builtin_ntp_servers=127.4.177.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:56.880574 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:56.880803 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:56.880841 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:56.881914 26333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:56.881973 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:56.882028 26333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.66
I20260808 21:09:56.883436 26333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.177.66
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.66
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:56.883703 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:56.884012 26333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:56.888942 26339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.888984 26333 server_base.cc:1034] running on GCE node
W20260808 21:09:56.888909 26340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:56.888895 26342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:56.889369 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:56.889624 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:56.890770 26333 hybrid_clock.cc:648] HybridClock initialized: now 1786223396890759 us; error 29 us; skew 500 ppm
I20260808 21:09:56.892045 26333 webserver.cc:460] Webserver started at http://127.4.177.66:41639/ using document root <none> and password file <none>
I20260808 21:09:56.892285 26333 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:56.892355 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:56.892522 26333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:56.893620 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data/instance:
uuid: "f12a86c7787e4e1ab04a11e9911d77ed"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "5675a74d5717391c55dd53982576cde3"
server_key_iv: "f5cec00b9504190e5ef2bad1da739612"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.893990 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance:
uuid: "f12a86c7787e4e1ab04a11e9911d77ed"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "5675a74d5717391c55dd53982576cde3"
server_key_iv: "f5cec00b9504190e5ef2bad1da739612"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.895354 26333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:56.896095 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.896273 26333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:56.896380 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal
uuid: "f12a86c7787e4e1ab04a11e9911d77ed"
format_stamp: "Formatted at 2026-08-08 21:09:56 on dist-test-slave-chsk"
server_key: "5675a74d5717391c55dd53982576cde3"
server_key_iv: "f5cec00b9504190e5ef2bad1da739612"
server_key_version: "encryptionkey@0"
I20260808 21:09:56.896452 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:56.937908 26333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:56.938246 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:56.938412 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:56.938666 26333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:56.939006 26333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:56.939062 26333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.939095 26333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:56.939124 26333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:56.945050 26333 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.66:43453
I20260808 21:09:56.945108 26473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.66:43453 every 8 connection(s)
I20260808 21:09:56.945438 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/data/info.pb
I20260808 21:09:56.950415 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.946245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.66:35573 (local address 127.4.177.126:44271)
0808 21:09:56.946294 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:56.946296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:56.946550 (+ 254us) server_negotiation.cc:407] Connection header received
0808 21:09:56.947045 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:56.947046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:56.947056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:56.947116 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:56.947798 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.948285 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.949210 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.949391 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.949442 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.950183 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:56.950184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:56.950185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:56.950189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:56.950203 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:56.950207 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:56.950256 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:56.950335 (+ 79us) server_negotiation.cc:299] Negotiation successful
0808 21:09:56.950356 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:09:56.950600 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:56.946202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44271 (local address 127.4.177.66:35573)
0808 21:09:56.946413 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:56.946455 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:56.946959 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:56.947195 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:56.947198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:56.947206 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:56.947710 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.947714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.948360 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.948361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.949134 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:56.949136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:56.949464 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:56.949465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:56.949568 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:56.950054 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:56.950054 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:56.950081 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:56.950102 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:56.950264 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:56.950268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:56.950271 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:56.950322 (+ 51us) client_negotiation.cc:240] Negotiation successful
0808 21:09:56.950399 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260808 21:09:56.951107 26474 heartbeater.cc:347] Connected to a master server at 127.4.177.126:44271
I20260808 21:09:56.951236 26474 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:56.951416 26474 heartbeater.cc:511] Master 127.4.177.126:44271 requested a full tablet report, sending...
I20260808 21:09:56.951808 26124 ts_manager.cc:194] Registered new tserver with Master: f12a86c7787e4e1ab04a11e9911d77ed (127.4.177.66:43453)
I20260808 21:09:56.952164 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.66:35573
I20260808 21:09:56.954278 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26333
I20260808 21:09:56.954370 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal/instance
I20260808 21:09:56.954519 4805 external_mini_cluster.cc:1428] Setting key 7c5f8d677d3d13367ff779b20f5ce7c9
I20260808 21:09:56.955417 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.177.67:0
--local_ip_for_outbound_sockets=127.4.177.67
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--builtin_ntp_servers=127.4.177.84:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 21:09:57.036535 26477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:57.036758 26477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:57.036798 26477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:57.037922 26477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:57.037985 26477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:57.038026 26477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.177.67
I20260808 21:09:57.039471 26477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.177.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.177.67
--webserver_port=0
--tserver_master_addrs=127.4.177.126:44271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.177.67
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:57.039716 26477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:57.040006 26477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:57.044899 26483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:57.044945 26477 server_base.cc:1034] running on GCE node
W20260808 21:09:57.044878 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:57.044867 26485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:57.045298 26477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:57.045540 26477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:57.046712 26477 hybrid_clock.cc:648] HybridClock initialized: now 1786223397046701 us; error 26 us; skew 500 ppm
I20260808 21:09:57.047995 26477 webserver.cc:460] Webserver started at http://127.4.177.67:33533/ using document root <none> and password file <none>
I20260808 21:09:57.048249 26477 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:57.048322 26477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:57.048506 26477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:57.049551 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data/instance:
uuid: "026b6ebb0c4c4720926d75dc5c1715a4"
format_stamp: "Formatted at 2026-08-08 21:09:57 on dist-test-slave-chsk"
server_key: "13ad21e0400b606d760bad63400893b7"
server_key_iv: "5691d885f987a37253ce3837d4aadaa8"
server_key_version: "encryptionkey@0"
I20260808 21:09:57.049940 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance:
uuid: "026b6ebb0c4c4720926d75dc5c1715a4"
format_stamp: "Formatted at 2026-08-08 21:09:57 on dist-test-slave-chsk"
server_key: "13ad21e0400b606d760bad63400893b7"
server_key_iv: "5691d885f987a37253ce3837d4aadaa8"
server_key_version: "encryptionkey@0"
I20260808 21:09:57.051275 26477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:09:57.051991 26493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:57.052166 26477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:09:57.052270 26477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal
uuid: "026b6ebb0c4c4720926d75dc5c1715a4"
format_stamp: "Formatted at 2026-08-08 21:09:57 on dist-test-slave-chsk"
server_key: "13ad21e0400b606d760bad63400893b7"
server_key_iv: "5691d885f987a37253ce3837d4aadaa8"
server_key_version: "encryptionkey@0"
I20260808 21:09:57.052366 26477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:57.066720 26477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:57.066998 26477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:57.067198 26477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:57.067428 26477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 21:09:57.067740 26477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 21:09:57.067807 26477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:57.067864 26477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 21:09:57.067891 26477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:57.073691 26477 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.67:32823
I20260808 21:09:57.073789 26615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.67:32823 every 8 connection(s)
I20260808 21:09:57.074079 26477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/data/info.pb
I20260808 21:09:57.078779 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.074758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.67:60983 (local address 127.4.177.126:44271)
0808 21:09:57.074801 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.074803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.075044 (+ 241us) server_negotiation.cc:407] Connection header received
0808 21:09:57.075578 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.075579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.075589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.075646 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:57.076245 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.076713 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.077616 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.077796 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.077845 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.078556 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:57.078557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:57.078557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:57.078561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:57.078574 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:57.078578 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:57.078620 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.078700 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.078719 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:09:57.078974 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.074729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44271 (local address 127.4.177.67:60983)
0808 21:09:57.074915 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.074954 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.075491 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.075695 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.075699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.075706 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:09:57.076174 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.076177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.076791 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.076791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.077543 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.077545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.077866 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.077866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.077967 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.078431 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 21:09:57.078431 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:09:57.078457 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 21:09:57.078479 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:09:57.078632 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:09:57.078635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:09:57.078638 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.078688 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.078762 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260808 21:09:57.079501 26616 heartbeater.cc:347] Connected to a master server at 127.4.177.126:44271
I20260808 21:09:57.079591 26616 heartbeater.cc:464] Registering TS with master...
I20260808 21:09:57.079725 26616 heartbeater.cc:511] Master 127.4.177.126:44271 requested a full tablet report, sending...
I20260808 21:09:57.080067 26124 ts_manager.cc:194] Registered new tserver with Master: 026b6ebb0c4c4720926d75dc5c1715a4 (127.4.177.67:32823)
I20260808 21:09:57.080446 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.177.67:60983
I20260808 21:09:57.080824 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26477
I20260808 21:09:57.080942 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal/instance
I20260808 21:09:57.081146 4805 external_mini_cluster.cc:1428] Setting key 39870bca6a214a475c2187496a22b99d
I20260808 21:09:57.082002 4805 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 21:09:57.086292 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.083507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.4.177.126:44271)
0808 21:09:57.083540 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.083542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.083710 (+ 168us) server_negotiation.cc:407] Connection header received
0808 21:09:57.083787 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.083788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.083796 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.083841 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:57.084017 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.084467 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.085214 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.085434 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.085507 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.086062 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:57.086064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:57.086064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:57.086069 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:57.086081 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:57.086084 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:57.086138 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.086197 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.086216 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 21:09:57.087652 26124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 21:09:57.089130 26124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 21:09:57.095604 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.4.177.66:43453)
0808 21:09:57.091659 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.091662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.091672 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:09:57.091731 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.091733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.091749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.091845 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:57.092016 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.092976 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.094495 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.095245 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.095327 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.095411 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.095513 (+ 102us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.095545 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260808 21:09:57.095607 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:43453 (local address 127.0.0.1:56480)
0808 21:09:57.091225 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.091274 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.091337 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.091866 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.091869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.091871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:57.091930 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.091931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.093058 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.093059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.094403 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.094406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.095333 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.095334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.095408 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.095421 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.095494 (+ 73us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.095540 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
I20260808 21:09:57.096035 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:32823 (local address 127.0.0.1:58194)
0808 21:09:57.091453 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.091465 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.091516 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.092059 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.092060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.092061 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:57.092107 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.092108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.093352 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.093353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.094403 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.094406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.095814 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.095815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.095872 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.095875 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.095930 (+ 55us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.095969 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":83,"threads_started":1}
I20260808 21:09:57.096068 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.4.177.67:32823)
0808 21:09:57.091788 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.091792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.091799 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:09:57.091850 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.091852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.091866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.092006 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:57.092178 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.093238 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.094493 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.095231 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.095327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.095411 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.095959 (+ 548us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.095997 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":75,"threads_started":1}
I20260808 21:09:57.096401 26401 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 (DEFAULT_TABLE table=test-table [id=d5c3e7bd663c48e2b39e4a68854d70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:57.096652 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 (DEFAULT_TABLE table=test-table [id=d5c3e7bd663c48e2b39e4a68854d70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:57.096810 26401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7c579be1948e48dcdd6b24f9b7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:57.096971 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7c579be1948e48dcdd6b24f9b7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:57.099406 26632 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Bootstrap starting.
I20260808 21:09:57.099718 26633 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Bootstrap starting.
I20260808 21:09:57.100095 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:40551 (local address 127.0.0.1:48224)
0808 21:09:57.091651 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.091658 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.091702 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.092021 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.092022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.092023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:57.092056 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.092058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.094506 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.094507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.095241 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.095243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.099859 (+ 4616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.099860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.099918 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.099922 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.099982 (+ 60us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.100020 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":74,"threads_started":1}
I20260808 21:09:57.100112 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.091425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.4.177.65:40551)
0808 21:09:57.091603 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.091608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.091713 (+ 105us) server_negotiation.cc:407] Connection header received
0808 21:09:57.091754 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.091756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.091769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.091845 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:57.092155 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.093229 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.095367 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.099732 (+ 4365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.099821 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.099906 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.099995 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.100031 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260808 21:09:57.100154 26632 tablet_bootstrap.cc:654] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:57.100340 26633 tablet_bootstrap.cc:654] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:57.100490 26632 log.cc:824] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:57.100629 26633 log.cc:824] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:57.101016 26258 tablet_service.cc:1467] Processing CreateTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 (DEFAULT_TABLE table=test-table [id=d5c3e7bd663c48e2b39e4a68854d70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 21:09:57.101258 26632 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: No bootstrap required, opened a new log
I20260808 21:09:57.101302 26632 ts_tablet_manager.cc:1397] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:57.101369 26258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b7c579be1948e48dcdd6b24f9b7291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:57.101389 26633 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: No bootstrap required, opened a new log
I20260808 21:09:57.101440 26633 ts_tablet_manager.cc:1397] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:57.103012 26632 raft_consensus.cc:348] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.103122 26632 raft_consensus.cc:374] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:57.103202 26632 raft_consensus.cc:729] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12a86c7787e4e1ab04a11e9911d77ed, State: Initialized, Role: FOLLOWER
I20260808 21:09:57.103334 26632 consensus_queue.cc:260] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.103613 26632 ts_tablet_manager.cc:1428] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 21:09:57.103634 26474 heartbeater.cc:503] Master 127.4.177.126:44271 was elected leader, sending a full tablet report...
I20260808 21:09:57.104454 26633 raft_consensus.cc:348] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.104614 26633 raft_consensus.cc:374] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:57.104664 26633 raft_consensus.cc:729] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026b6ebb0c4c4720926d75dc5c1715a4, State: Initialized, Role: FOLLOWER
I20260808 21:09:57.104789 26633 consensus_queue.cc:260] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.104914 26638 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: Bootstrap starting.
I20260808 21:09:57.105072 26633 ts_tablet_manager.cc:1428] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260808 21:09:57.105083 26616 heartbeater.cc:503] Master 127.4.177.126:44271 was elected leader, sending a full tablet report...
I20260808 21:09:57.105562 26638 tablet_bootstrap.cc:654] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:57.105885 26638 log.cc:824] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:57.106544 26638 tablet_bootstrap.cc:492] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: No bootstrap required, opened a new log
I20260808 21:09:57.106595 26638 ts_tablet_manager.cc:1397] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260808 21:09:57.108431 26638 raft_consensus.cc:348] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.108543 26638 raft_consensus.cc:374] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:57.108564 26638 raft_consensus.cc:729] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea5a6aed16444d0a0dddcdbaa3f7ca7, State: Initialized, Role: FOLLOWER
I20260808 21:09:57.108716 26638 consensus_queue.cc:260] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.108966 26638 ts_tablet_manager.cc:1428] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 21:09:57.109112 26330 heartbeater.cc:503] Master 127.4.177.126:44271 was elected leader, sending a full tablet report...
I20260808 21:09:57.160956 26641 raft_consensus.cc:482] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 21:09:57.161085 26641 raft_consensus.cc:504] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.161433 26641 leader_election.cc:288] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f12a86c7787e4e1ab04a11e9911d77ed (127.4.177.66:43453), 026b6ebb0c4c4720926d75dc5c1715a4 (127.4.177.67:32823)
I20260808 21:09:57.164354 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.161582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.66:43453 (local address 127.4.177.65:55233)
0808 21:09:57.161632 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.161643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.161679 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.161832 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.161833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.161834 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:57.161902 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.161903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.162553 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.162554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.163538 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.163540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.164100 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.164101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.164177 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.164180 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.164244 (+ 64us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.164284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:09:57.164354 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.161631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:55233 (local address 127.4.177.66:43453)
0808 21:09:57.161704 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.161706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.161715 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:09:57.161740 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.161741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.161751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.161818 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:57.161973 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.162465 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.163621 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.164015 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.164101 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.164109 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.164250 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.164292 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 21:09:57.164624 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.161591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.67:32823 (local address 127.4.177.65:44629)
0808 21:09:57.161732 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.161751 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 21:09:57.161827 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:09:57.161963 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:09:57.161964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:09:57.161965 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 21:09:57.162048 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.162050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.162667 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.162668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.163693 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:09:57.163696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.164305 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.164306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:09:57.164386 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.164392 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 21:09:57.164503 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 21:09:57.164553 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260808 21:09:57.164644 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.161655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.177.65:44629 (local address 127.4.177.67:32823)
0808 21:09:57.161696 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.161698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.161818 (+ 120us) server_negotiation.cc:407] Connection header received
0808 21:09:57.161884 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.161885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.161899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.161959 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 21:09:57.162138 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.162581 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.163786 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.164206 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.164302 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.164310 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.164539 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.164581 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:09:57.164662 26422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7c579be1948e48dcdd6b24f9b7291" candidate_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" is_pre_election: true
I20260808 21:09:57.164788 26422 raft_consensus.cc:2454] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea5a6aed16444d0a0dddcdbaa3f7ca7 in term 0.
I20260808 21:09:57.164989 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7c579be1948e48dcdd6b24f9b7291" candidate_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" is_pre_election: true
I20260808 21:09:57.165072 26209 leader_election.cc:302] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea5a6aed16444d0a0dddcdbaa3f7ca7, f12a86c7787e4e1ab04a11e9911d77ed; no voters:
I20260808 21:09:57.165122 26565 raft_consensus.cc:2454] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea5a6aed16444d0a0dddcdbaa3f7ca7 in term 0.
I20260808 21:09:57.165201 26641 raft_consensus.cc:2781] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 21:09:57.165292 26641 raft_consensus.cc:482] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 21:09:57.165319 26641 raft_consensus.cc:3037] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:57.165987 26641 raft_consensus.cc:504] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.166123 26641 leader_election.cc:288] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [CANDIDATE]: Term 1 election: Requested vote from peers f12a86c7787e4e1ab04a11e9911d77ed (127.4.177.66:43453), 026b6ebb0c4c4720926d75dc5c1715a4 (127.4.177.67:32823)
I20260808 21:09:57.166275 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7c579be1948e48dcdd6b24f9b7291" candidate_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026b6ebb0c4c4720926d75dc5c1715a4"
I20260808 21:09:57.166277 26422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b7c579be1948e48dcdd6b24f9b7291" candidate_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12a86c7787e4e1ab04a11e9911d77ed"
I20260808 21:09:57.166344 26422 raft_consensus.cc:3037] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:57.166359 26565 raft_consensus.cc:3037] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:57.167027 26565 raft_consensus.cc:2454] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea5a6aed16444d0a0dddcdbaa3f7ca7 in term 1.
I20260808 21:09:57.167213 26422 raft_consensus.cc:2454] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea5a6aed16444d0a0dddcdbaa3f7ca7 in term 1.
I20260808 21:09:57.167213 26208 leader_election.cc:302] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026b6ebb0c4c4720926d75dc5c1715a4, cea5a6aed16444d0a0dddcdbaa3f7ca7; no voters:
I20260808 21:09:57.167341 26641 raft_consensus.cc:2781] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:57.167490 26641 raft_consensus.cc:686] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 1 LEADER]: Becoming Leader. State: Replica: cea5a6aed16444d0a0dddcdbaa3f7ca7, State: Running, Role: LEADER
I20260808 21:09:57.167591 26641 consensus_queue.cc:237] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } }
I20260808 21:09:57.168401 26124 catalog_manager.cc:5495] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cea5a6aed16444d0a0dddcdbaa3f7ca7 (127.4.177.65). New cstate: current_term: 1 leader_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea5a6aed16444d0a0dddcdbaa3f7ca7" member_type: VOTER last_known_addr { host: "127.4.177.65" port: 40551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 } health_report { overall_health: UNKNOWN } } }
I20260808 21:09:57.185897 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.183569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.4.177.65:40551)
0808 21:09:57.183606 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:57.183608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:57.183616 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:57.183706 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:57.183707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:57.183724 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:57.183801 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:09:57.183930 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.184366 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.185100 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:57.185310 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:57.185385 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:57.185542 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:09:57.185646 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:09:57.185737 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:57.185774 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 21:09:57.185792 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:09:57.187494 26422 raft_consensus.cc:1261] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 1 FOLLOWER]: Refusing update from remote peer cea5a6aed16444d0a0dddcdbaa3f7ca7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:57.187515 26565 raft_consensus.cc:1261] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 1 FOLLOWER]: Refusing update from remote peer cea5a6aed16444d0a0dddcdbaa3f7ca7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:09:57.187827 26641 consensus_queue.cc:1035] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12a86c7787e4e1ab04a11e9911d77ed" member_type: VOTER last_known_addr { host: "127.4.177.66" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:57.188025 26644 consensus_queue.cc:1035] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026b6ebb0c4c4720926d75dc5c1715a4" member_type: VOTER last_known_addr { host: "127.4.177.67" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:09:57.190810 26649 mvcc.cc:205] Tried to move back new op lower bound from 7316371034877411328 to 7316371034799149056. Current Snapshot: MvccSnapshot[applied={T|T < 7316371034877411328}]
I20260808 21:09:57.192749 26124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:57.192857 26124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 21:09:57.193569 26124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49f665b0694846799170e8d1650a897e: Sending DeleteTablet for 3 replicas of tablet 63b7c579be1948e48dcdd6b24f9b7291
I20260808 21:09:57.193921 26401 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:57 UTC) from {username='slave'} at 127.0.0.1:56480
I20260808 21:09:57.193920 26544 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:57 UTC) from {username='slave'} at 127.0.0.1:58194
I20260808 21:09:57.194185 26258 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b7c579be1948e48dcdd6b24f9b7291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 21:09:57 UTC) from {username='slave'} at 127.0.0.1:48224
I20260808 21:09:57.194389 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26190
I20260808 21:09:57.194404 26660 tablet_replica.cc:330] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7: stopping tablet replica
I20260808 21:09:57.194502 26660 raft_consensus.cc:2229] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 1 LEADER]: Raft consensus shutting down.
I20260808 21:09:57.194592 26660 raft_consensus.cc:2258] T 63b7c579be1948e48dcdd6b24f9b7291 P cea5a6aed16444d0a0dddcdbaa3f7ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:57.199748 26657 tablet_replica.cc:330] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: stopping tablet replica
I20260808 21:09:57.199851 26657 raft_consensus.cc:2229] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:57.199965 26657 raft_consensus.cc:2258] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:57.200116 26659 tablet_replica.cc:330] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: stopping tablet replica
I20260808 21:09:57.200212 26659 raft_consensus.cc:2229] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 21:09:57.200328 26659 raft_consensus.cc:2258] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 21:09:57.200415 26657 ts_tablet_manager.cc:1905] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:57.200723 26659 ts_tablet_manager.cc:1905] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 21:09:57.202096 26657 ts_tablet_manager.cc:1918] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:57.202184 26657 log.cc:1196] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-1/wal/wals/63b7c579be1948e48dcdd6b24f9b7291
I20260808 21:09:57.202484 26657 ts_tablet_manager.cc:1939] T 63b7c579be1948e48dcdd6b24f9b7291 P f12a86c7787e4e1ab04a11e9911d77ed: Deleting consensus metadata
I20260808 21:09:57.202646 26659 ts_tablet_manager.cc:1918] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 21:09:57.202723 26659 log.cc:1196] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Deleting WAL directory at /tmp/dist-test-task496WfL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786223155551886-4805-0/minicluster-data/ts-2/wal/wals/63b7c579be1948e48dcdd6b24f9b7291
I20260808 21:09:57.202965 26659 ts_tablet_manager.cc:1939] T 63b7c579be1948e48dcdd6b24f9b7291 P 026b6ebb0c4c4720926d75dc5c1715a4: Deleting consensus metadata
I20260808 21:09:57.202955 26108 catalog_manager.cc:4841] TS f12a86c7787e4e1ab04a11e9911d77ed (127.4.177.66:43453): tablet 63b7c579be1948e48dcdd6b24f9b7291 (table test-table [id=d5c3e7bd663c48e2b39e4a68854d70ac]) successfully deleted
I20260808 21:09:57.203361 26107 catalog_manager.cc:4841] TS 026b6ebb0c4c4720926d75dc5c1715a4 (127.4.177.67:32823): tablet 63b7c579be1948e48dcdd6b24f9b7291 (table test-table [id=d5c3e7bd663c48e2b39e4a68854d70ac]) successfully deleted
W20260808 21:09:57.204299 26108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.177.65:40551 (error 108)
I20260808 21:09:57.204779 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:57.204625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.65:40551 (local address 127.0.0.1:48244)
0808 21:09:57.204680 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0808 21:09:57.204722 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.65:40551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 21:09:57.204849 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26333
W20260808 21:09:57.204923 26108 catalog_manager.cc:4584] TS cea5a6aed16444d0a0dddcdbaa3f7ca7 (127.4.177.65:40551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63b7c579be1948e48dcdd6b24f9b7291: Network error: Client connection negotiation failed: client connection to 127.4.177.65:40551: connect: Connection refused (error 111)
I20260808 21:09:57.210687 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26477
I20260808 21:09:57.216389 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26090
2026-08-08T21:09:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (743 ms)
[----------] 6 tests from EncryptionPolicyTest (8701 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:57 dist-test-slave-chsk krb5kdc[26664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:57 dist-test-slave-chsk krb5kdc[26664](info): set up 2 sockets
Aug 08 21:09:57 dist-test-slave-chsk krb5kdc[26664](info): commencing operation
krb5kdc: starting...
W20260808 21:09:58.282348 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:09:58.402616 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:58.483965 26677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:58.484165 26677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:58.484189 26677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:58.485342 26677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:58.485387 26677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:58.485402 26677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:58.485447 26677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:58.485466 26677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:58.486933 26677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:58.487200 26677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:58.487453 26677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:58.492203 26685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:58.492204 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:58.492225 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:58.492393 26677 server_base.cc:1034] running on GCE node
I20260808 21:09:58.492637 26677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:58.492934 26677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:58.494100 26677 hybrid_clock.cc:648] HybridClock initialized: now 1786223398494098 us; error 40 us; skew 500 ppm
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:58.496565 26677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:09:58.496984 26677 webserver.cc:460] Webserver started at http://127.0.0.1:43667/ using document root <none> and password file <none>
I20260808 21:09:58.497226 26677 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:58.497334 26677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:58.497501 26677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:58.498494 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "64af042652a548f9a04fd4dba0165bf1"
format_stamp: "Formatted at 2026-08-08 21:09:58 on dist-test-slave-chsk"
server_key: "cf278b78b9c47fe6665d549f439089c7"
server_key_iv: "2145eee0477695a4105d5a30bfca098f"
server_key_version: "encryptionkey@0"
I20260808 21:09:58.498852 26677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "64af042652a548f9a04fd4dba0165bf1"
format_stamp: "Formatted at 2026-08-08 21:09:58 on dist-test-slave-chsk"
server_key: "cf278b78b9c47fe6665d549f439089c7"
server_key_iv: "2145eee0477695a4105d5a30bfca098f"
server_key_version: "encryptionkey@0"
I20260808 21:09:58.500286 26677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:09:58.501008 26692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:58.501216 26677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:58.501318 26677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "64af042652a548f9a04fd4dba0165bf1"
format_stamp: "Formatted at 2026-08-08 21:09:58 on dist-test-slave-chsk"
server_key: "cf278b78b9c47fe6665d549f439089c7"
server_key_iv: "2145eee0477695a4105d5a30bfca098f"
server_key_version: "encryptionkey@0"
I20260808 21:09:58.501400 26677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:58.521631 26677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:58.523445 26677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:58.523607 26677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:58.527631 26677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34049
I20260808 21:09:58.527679 26748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34049 every 8 connection(s)
I20260808 21:09:58.528019 26677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:58.528957 26749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:58.531307 26749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1: Bootstrap starting.
I20260808 21:09:58.531894 26749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:58.532210 26749 log.cc:824] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:58.532883 26749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1: No bootstrap required, opened a new log
I20260808 21:09:58.534472 26749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260808 21:09:58.534603 26749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:58.534662 26749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64af042652a548f9a04fd4dba0165bf1, State: Initialized, Role: FOLLOWER
I20260808 21:09:58.534816 26749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260808 21:09:58.534875 26749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:58.534919 26749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:58.534967 26749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:58.535615 26749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260808 21:09:58.535730 26749 leader_election.cc:302] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64af042652a548f9a04fd4dba0165bf1; no voters:
I20260808 21:09:58.535887 26749 leader_election.cc:288] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:58.535910 26752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:58.536087 26752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 64af042652a548f9a04fd4dba0165bf1, State: Running, Role: LEADER
I20260808 21:09:58.536161 26749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:58.536171 26752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260808 21:09:58.536681 26754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64af042652a548f9a04fd4dba0165bf1. Latest consensus state: current_term: 1 leader_uuid: "64af042652a548f9a04fd4dba0165bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260808 21:09:58.536765 26754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:58.536815 26753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64af042652a548f9a04fd4dba0165bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64af042652a548f9a04fd4dba0165bf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260808 21:09:58.536916 26753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:58.536996 26760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:58.537528 26760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:58.537881 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26677
I20260808 21:09:58.538009 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:58.538220 4805 external_mini_cluster.cc:1428] Setting key e50da15293ee55cc4c777eb569baa3ed
I20260808 21:09:58.539108 26760 catalog_manager.cc:1347] Generated new cluster ID: c44f2ca861144a9495f1b460fa8d834d
I20260808 21:09:58.539181 26760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 21:09:58.550585 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:58.538855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44897 (local address 127.0.0.1:34049)
0808 21:09:58.539019 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:58.539028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:58.539058 (+ 30us) server_negotiation.cc:407] Connection header received
0808 21:09:58.539255 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:58.539261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:58.539317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:58.539469 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:58.540364 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:58.541853 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:58.542974 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:58.543617 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:58.543738 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:58.546706 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:58.546710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:58.546711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:58.546723 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:58.548750 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:58.549191 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:58.549194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:58.549194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:58.549318 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:58.549548 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:58.549549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:58.549550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:58.549848 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:58.550000 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:58.550105 (+ 105us) server_negotiation.cc:299] Negotiation successful
0808 21:09:58.550334 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260808 21:09:58.551640 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:09:58.556896 26760 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:58.557576 26760 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:58.560042 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:58.553092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58261 (local address 127.0.0.1:34049)
0808 21:09:58.553124 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:58.553126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:58.553243 (+ 117us) server_negotiation.cc:407] Connection header received
0808 21:09:58.553331 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:58.553332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:58.553340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:58.553367 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:58.554048 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:58.554803 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:58.555825 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:58.556125 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:58.556163 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:58.557488 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:58.557489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:58.557489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:09:58.557496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:58.558846 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:58.559271 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:58.559272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:58.559272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:58.559328 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:09:58.559546 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:09:58.559547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:09:58.559547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:09:58.559648 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:58.559773 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:58.559873 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 21:09:58.559933 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:09:58.566995 26760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64af042652a548f9a04fd4dba0165bf1: Generated new TSK 0
I20260808 21:09:58.567210 26760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:09:58.569937 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26677
W20260808 21:09:58.576521 26662 connection.cc:535] client connection to 127.0.0.1:34049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34049): Connection reset by peer (error 104)
2026-08-08T21:09:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26782](info): set up 2 sockets
Aug 08 21:09:58 dist-test-slave-chsk krb5kdc[26782](info): commencing operation
krb5kdc: starting...
W20260808 21:09:59.647568 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:09:59 dist-test-slave-chsk krb5kdc[26782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:09:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:09:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:09:59.782505 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:09:59.864681 26798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:09:59.864887 26798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:09:59.864910 26798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:09:59.865978 26798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:09:59.866014 26798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:09:59.866029 26798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:09:59.866041 26798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:09:59.866053 26798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:09:59.867496 26798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:09:59.867690 26798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:09:59.867859 26798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:09:59.872596 26803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:59.872596 26804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:09:59.872610 26806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:09:59.872694 26798 server_base.cc:1034] running on GCE node
I20260808 21:09:59.873085 26798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:09:59.873409 26798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:09:59.874580 26798 hybrid_clock.cc:648] HybridClock initialized: now 1786223399874573 us; error 36 us; skew 500 ppm
Aug 08 21:09:59 dist-test-slave-chsk krb5kdc[26782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:09:59.877338 26798 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:09:59.877841 26798 webserver.cc:460] Webserver started at http://127.0.0.1:44691/ using document root <none> and password file <none>
I20260808 21:09:59.878098 26798 fs_manager.cc:359] Metadata directory not provided
I20260808 21:09:59.878170 26798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:09:59.878326 26798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:09:59.879493 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "34234c5567804b2cb3346a94fe81ca92"
format_stamp: "Formatted at 2026-08-08 21:09:59 on dist-test-slave-chsk"
server_key: "e320a018c01006b6712c390e69b559ac"
server_key_iv: "61de3a5be7a339d8fb3a820a5936dd66"
server_key_version: "encryptionkey@0"
I20260808 21:09:59.879856 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "34234c5567804b2cb3346a94fe81ca92"
format_stamp: "Formatted at 2026-08-08 21:09:59 on dist-test-slave-chsk"
server_key: "e320a018c01006b6712c390e69b559ac"
server_key_iv: "61de3a5be7a339d8fb3a820a5936dd66"
server_key_version: "encryptionkey@0"
I20260808 21:09:59.881664 26798 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260808 21:09:59.882455 26813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:09:59.882634 26798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:09:59.882735 26798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "34234c5567804b2cb3346a94fe81ca92"
format_stamp: "Formatted at 2026-08-08 21:09:59 on dist-test-slave-chsk"
server_key: "e320a018c01006b6712c390e69b559ac"
server_key_iv: "61de3a5be7a339d8fb3a820a5936dd66"
server_key_version: "encryptionkey@0"
I20260808 21:09:59.882822 26798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:09:59.898988 26798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:09:59.901240 26798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:09:59.901397 26798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:09:59.905650 26798 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35135
I20260808 21:09:59.905683 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35135 every 8 connection(s)
I20260808 21:09:59.906013 26798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:09:59.906984 26867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:09:59.908115 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26798
I20260808 21:09:59.908236 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:09:59.908463 4805 external_mini_cluster.cc:1428] Setting key c90a8a32ea3a2c9c5b061324439f7386
I20260808 21:09:59.909829 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92: Bootstrap starting.
I20260808 21:09:59.910460 26867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92: Neither blocks nor log segments found. Creating new log.
I20260808 21:09:59.910784 26867 log.cc:824] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92: Log is configured to *not* fsync() on all Append() calls
I20260808 21:09:59.911545 26867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92: No bootstrap required, opened a new log
I20260808 21:09:59.913913 26867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260808 21:09:59.914083 26867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:09:59.914129 26867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34234c5567804b2cb3346a94fe81ca92, State: Initialized, Role: FOLLOWER
I20260808 21:09:59.914341 26867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260808 21:09:59.914407 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:59.908997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48117 (local address 127.0.0.1:35135)
0808 21:09:59.909197 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:59.909204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:59.909228 (+ 24us) server_negotiation.cc:407] Connection header received
0808 21:09:59.909649 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:59.909655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:59.909699 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:59.909822 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:59.910243 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:59.911430 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:59.912501 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:59.912912 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:59.912992 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:59.913733 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:59.913737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:59.913737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:59.913896 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:59.913923 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:59.913933 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:59.914007 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:59.914167 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 21:09:59.914198 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260808 21:09:59.914424 26867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:09:59.914565 26867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:09:59.914623 26867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:09:59.915340 26867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260808 21:09:59.915446 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:09:59.915448 26867 leader_election.cc:302] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34234c5567804b2cb3346a94fe81ca92; no voters:
I20260808 21:09:59.915606 26867 leader_election.cc:288] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:09:59.915665 26873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:09:59.915823 26873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [term 1 LEADER]: Becoming Leader. State: Replica: 34234c5567804b2cb3346a94fe81ca92, State: Running, Role: LEADER
I20260808 21:09:59.915863 26867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:09:59.915961 26873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } }
I20260808 21:09:59.916286 26873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34234c5567804b2cb3346a94fe81ca92. Latest consensus state: current_term: 1 leader_uuid: "34234c5567804b2cb3346a94fe81ca92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260808 21:09:59.916383 26873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:59.916297 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34234c5567804b2cb3346a94fe81ca92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34234c5567804b2cb3346a94fe81ca92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35135 } } }
I20260808 21:09:59.916522 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92 [sys.catalog]: This master's current role is: LEADER
I20260808 21:09:59.916613 26881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:09:59.917135 26881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:09:59.918560 26881 catalog_manager.cc:1347] Generated new cluster ID: 4e96e7bc726e45d9b92b02d561588e41
I20260808 21:09:59.918625 26881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:09:59.923906 26881 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:09:59.924432 26881 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:09:59.926219 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:09:59.922838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54609 (local address 127.0.0.1:35135)
0808 21:09:59.922902 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0808 21:09:59.922903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:09:59.922911 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:09:59.922928 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:09:59.922929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:09:59.922939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:09:59.922990 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:09:59.923250 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:59.923903 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:59.924914 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:09:59.925204 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:09:59.925268 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:09:59.925949 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:09:59.925950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:09:59.925950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:09:59.925954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:09:59.925968 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:09:59.925973 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:09:59.926030 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:09:59.926138 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 21:09:59.926148 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":43}
I20260808 21:09:59.928715 26881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34234c5567804b2cb3346a94fe81ca92: Generated new TSK 0
I20260808 21:09:59.928870 26881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:09:59.933030 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26798
W20260808 21:09:59.939771 26780 connection.cc:535] client connection to 127.0.0.1:35135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35135): Connection reset by peer (error 104)
2026-08-08T21:09:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:09:59 dist-test-slave-chsk krb5kdc[26900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:09:59 dist-test-slave-chsk krb5kdc[26900](info): set up 2 sockets
Aug 08 21:09:59 dist-test-slave-chsk krb5kdc[26900](info): commencing operation
krb5kdc: starting...
W20260808 21:10:01.023248 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:01 dist-test-slave-chsk krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:01.146189 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:01.228145 26914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:01.228408 26914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:01.228451 26914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:01.229610 26914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:01.229679 26914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:01.229704 26914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:01.229725 26914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:01.229780 26914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:01.231324 26914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:01.231609 26914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:01.231844 26914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:01.236792 26923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:01.236933 26914 server_base.cc:1034] running on GCE node
W20260808 21:10:01.236826 26921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:01.237026 26920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:01.237270 26914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:01.237564 26914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:01.238711 26914 hybrid_clock.cc:648] HybridClock initialized: now 1786223401238698 us; error 30 us; skew 500 ppm
Aug 08 21:10:01 dist-test-slave-chsk krb5kdc[26900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:01.241251 26914 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:01.241675 26914 webserver.cc:460] Webserver started at http://127.0.0.1:43103/ using document root <none> and password file <none>
I20260808 21:10:01.241926 26914 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:01.241997 26914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:01.242152 26914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:01.243352 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "4bc15836430d430f8977981c91c1ce5f"
format_stamp: "Formatted at 2026-08-08 21:10:01 on dist-test-slave-chsk"
server_key: "ea16a5ea9540b4a498908f502ede3ff6"
server_key_iv: "73d2be3fc56a286e313e49fe18893a78"
server_key_version: "encryptionkey@0"
I20260808 21:10:01.243736 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "4bc15836430d430f8977981c91c1ce5f"
format_stamp: "Formatted at 2026-08-08 21:10:01 on dist-test-slave-chsk"
server_key: "ea16a5ea9540b4a498908f502ede3ff6"
server_key_iv: "73d2be3fc56a286e313e49fe18893a78"
server_key_version: "encryptionkey@0"
I20260808 21:10:01.245087 26914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:01.245903 26932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:01.246097 26914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:10:01.246210 26914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "4bc15836430d430f8977981c91c1ce5f"
format_stamp: "Formatted at 2026-08-08 21:10:01 on dist-test-slave-chsk"
server_key: "ea16a5ea9540b4a498908f502ede3ff6"
server_key_iv: "73d2be3fc56a286e313e49fe18893a78"
server_key_version: "encryptionkey@0"
I20260808 21:10:01.246294 26914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:01.249794 26914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:01.251971 26914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:01.252133 26914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:01.256330 26914 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41077
I20260808 21:10:01.256368 26989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41077 every 8 connection(s)
I20260808 21:10:01.256740 26914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:01.257733 26990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:01.260084 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f: Bootstrap starting.
I20260808 21:10:01.260684 26990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:01.260987 26990 log.cc:824] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:01.261372 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 26914
I20260808 21:10:01.261516 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:01.261750 4805 external_mini_cluster.cc:1428] Setting key c03c8fc0bf6a9e8eb2baa57a04f415dc
I20260808 21:10:01.261859 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f: No bootstrap required, opened a new log
I20260808 21:10:01.264055 26990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260808 21:10:01.264225 26990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:01.264274 26990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc15836430d430f8977981c91c1ce5f, State: Initialized, Role: FOLLOWER
I20260808 21:10:01.264426 26990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260808 21:10:01.264549 26990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:01.264595 26990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:01.264650 26990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:01.264750 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:01.262302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34999 (local address 127.0.0.1:41077)
0808 21:10:01.262489 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:01.262500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:01.262534 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:10:01.263190 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:01.263199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:01.263242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:01.263396 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:01.264070 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:01.264073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:01.264074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:01.264078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:01.264127 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:01.264137 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:01.264213 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:01.264285 (+ 72us) server_negotiation.cc:299] Negotiation successful
0808 21:10:01.264412 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260808 21:10:01.265338 26990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260808 21:10:01.265448 26990 leader_election.cc:302] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bc15836430d430f8977981c91c1ce5f; no voters:
I20260808 21:10:01.265611 26990 leader_election.cc:288] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:01.265661 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:01.265699 26995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:01.265844 26995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [term 1 LEADER]: Becoming Leader. State: Replica: 4bc15836430d430f8977981c91c1ce5f, State: Running, Role: LEADER
I20260808 21:10:01.265865 26990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:01.265936 26995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } }
I20260808 21:10:01.266206 26995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bc15836430d430f8977981c91c1ce5f. Latest consensus state: current_term: 1 leader_uuid: "4bc15836430d430f8977981c91c1ce5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } } }
I20260808 21:10:01.266217 26997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bc15836430d430f8977981c91c1ce5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bc15836430d430f8977981c91c1ce5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41077 } } }
I20260808 21:10:01.266300 26995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:01.266309 26997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:01.267467 27000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:01.268006 27000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:01.268663 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:01.267281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58017 (local address 127.0.0.1:41077)
0808 21:10:01.267319 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:01.267321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:01.267654 (+ 333us) server_negotiation.cc:407] Connection header received
0808 21:10:01.267728 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:01.267729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:01.267734 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:01.267816 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:01.268433 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:01.268435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:01.268435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:01.268440 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:01.268453 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:01.268457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:01.268514 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:01.268577 (+ 63us) server_negotiation.cc:299] Negotiation successful
0808 21:10:01.268590 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:10:01.269626 27000 catalog_manager.cc:1347] Generated new cluster ID: d33e72ad17e2472f959811bd59ce82f5
I20260808 21:10:01.269682 27000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:01.277932 27000 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:01.278096 27000 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:01.282600 27000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bc15836430d430f8977981c91c1ce5f: Generated new TSK 0
I20260808 21:10:01.282779 27000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:01.288388 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 26914
2026-08-08T21:10:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:01 dist-test-slave-chsk krb5kdc[27022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:01 dist-test-slave-chsk krb5kdc[27022](info): set up 2 sockets
Aug 08 21:10:01 dist-test-slave-chsk krb5kdc[27022](info): commencing operation
krb5kdc: starting...
W20260808 21:10:02.376961 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:02.501427 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:02.581470 27036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:02.581708 27036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:02.581748 27036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:02.582896 27036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:02.582963 27036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:02.582988 27036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:02.583010 27036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:02.583038 27036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:02.584496 27036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:02.584789 27036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:02.585005 27036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:02.589974 27045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:02.589956 27043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:02.589944 27042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:02.590197 27036 server_base.cc:1034] running on GCE node
I20260808 21:10:02.590437 27036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:02.590744 27036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:02.591893 27036 hybrid_clock.cc:648] HybridClock initialized: now 1786223402591861 us; error 43 us; skew 500 ppm
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:02.594277 27036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:02.594681 27036 webserver.cc:460] Webserver started at http://127.0.0.1:33191/ using document root <none> and password file <none>
I20260808 21:10:02.594901 27036 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:02.594969 27036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:02.595176 27036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:02.596189 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "f554e793237d49d48faee11ae19343be"
format_stamp: "Formatted at 2026-08-08 21:10:02 on dist-test-slave-chsk"
server_key: "bdde1960c007e2da80c50f66728cde8b"
server_key_iv: "9e9bc9ebc0d7d71db550c0b9b092a7a5"
server_key_version: "encryptionkey@0"
I20260808 21:10:02.596606 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "f554e793237d49d48faee11ae19343be"
format_stamp: "Formatted at 2026-08-08 21:10:02 on dist-test-slave-chsk"
server_key: "bdde1960c007e2da80c50f66728cde8b"
server_key_iv: "9e9bc9ebc0d7d71db550c0b9b092a7a5"
server_key_version: "encryptionkey@0"
I20260808 21:10:02.597889 27036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:02.598604 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:02.598773 27036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:02.598877 27036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "f554e793237d49d48faee11ae19343be"
format_stamp: "Formatted at 2026-08-08 21:10:02 on dist-test-slave-chsk"
server_key: "bdde1960c007e2da80c50f66728cde8b"
server_key_iv: "9e9bc9ebc0d7d71db550c0b9b092a7a5"
server_key_version: "encryptionkey@0"
I20260808 21:10:02.598950 27036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:02.611012 27036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:02.612730 27036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:02.612883 27036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:02.616850 27036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45027
I20260808 21:10:02.616911 27108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45027 every 8 connection(s)
I20260808 21:10:02.617247 27036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:02.618234 27109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:02.620530 27109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be: Bootstrap starting.
I20260808 21:10:02.621116 27109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:02.621402 27109 log.cc:824] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:02.622097 27109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be: No bootstrap required, opened a new log
I20260808 21:10:02.623739 27109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260808 21:10:02.623867 27109 raft_consensus.cc:374] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:02.623888 27109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554e793237d49d48faee11ae19343be, State: Initialized, Role: FOLLOWER
I20260808 21:10:02.624073 27109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260808 21:10:02.624136 27109 raft_consensus.cc:388] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:02.624159 27109 raft_consensus.cc:482] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:02.624224 27109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:02.624725 27109 raft_consensus.cc:504] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260808 21:10:02.624836 27109 leader_election.cc:302] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f554e793237d49d48faee11ae19343be; no voters:
I20260808 21:10:02.625020 27109 leader_election.cc:288] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:02.625073 27112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:02.625277 27112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [term 1 LEADER]: Becoming Leader. State: Replica: f554e793237d49d48faee11ae19343be, State: Running, Role: LEADER
I20260808 21:10:02.625279 27109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:02.625430 27112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } }
I20260808 21:10:02.625756 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27036
I20260808 21:10:02.625823 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f554e793237d49d48faee11ae19343be. Latest consensus state: current_term: 1 leader_uuid: "f554e793237d49d48faee11ae19343be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } } }
I20260808 21:10:02.625877 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:02.625898 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:02.625946 27113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f554e793237d49d48faee11ae19343be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f554e793237d49d48faee11ae19343be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45027 } } }
I20260808 21:10:02.626005 27113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:02.626103 4805 external_mini_cluster.cc:1428] Setting key 97f4334aea2dc8f0aaef254c58a6f4a1
I20260808 21:10:02.626667 27119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:02.627265 27119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:02.628760 27119 catalog_manager.cc:1347] Generated new cluster ID: 6206827f2ffa4d088a4807d9c32df49f
I20260808 21:10:02.628804 27119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 21:10:02.637852 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:02.626769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59107 (local address 127.0.0.1:45027)
0808 21:10:02.626998 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:02.627009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:02.627055 (+ 46us) server_negotiation.cc:407] Connection header received
0808 21:10:02.627197 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:02.627204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:02.627260 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:02.627443 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:02.628220 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:02.629657 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:02.630754 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:02.631379 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:02.631476 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:02.634195 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:02.634199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:02.634200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:02.634211 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:02.636185 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:02.636578 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:02.636581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:02.636582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:02.636705 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:02.636916 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:02.636917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:02.636917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:02.637227 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:02.637399 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:02.637506 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 21:10:02.637640 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260808 21:10:02.639029 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:02.643399 27119 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:02.644064 27119 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:02.647554 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:02.640514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46361 (local address 127.0.0.1:45027)
0808 21:10:02.640546 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:02.640547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:02.640621 (+ 74us) server_negotiation.cc:407] Connection header received
0808 21:10:02.640680 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:02.640681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:02.640690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:02.640756 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:02.641315 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:02.642098 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:02.643024 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:02.643349 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:02.643417 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:02.644758 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:02.644760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:02.644760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:02.644766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:02.646245 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:02.646582 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:02.646582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:02.646583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:02.646626 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:02.646785 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:02.646786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:02.646786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:02.646888 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:02.646960 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:02.647368 (+ 408us) server_negotiation.cc:299] Negotiation successful
0808 21:10:02.647468 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260808 21:10:02.648571 27119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f554e793237d49d48faee11ae19343be: Generated new TSK 0
I20260808 21:10:02.648729 27119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:02.650663 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27036
2026-08-08T21:10:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27143](info): set up 2 sockets
Aug 08 21:10:02 dist-test-slave-chsk krb5kdc[27143](info): commencing operation
krb5kdc: starting...
W20260808 21:10:03.754150 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:03 dist-test-slave-chsk krb5kdc[27143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:03.882535 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:03.964672 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:03.964882 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:03.964905 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:03.966050 27158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:03.966097 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:03.966111 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:03.966125 27158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:03.966140 27158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:03.967754 27158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:03.968017 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:03.968257 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:10:03.973239 27158 server_base.cc:1034] running on GCE node
W20260808 21:10:03.973217 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:03.973232 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:03.973210 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:03.973701 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:03.973995 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:03.975193 27158 hybrid_clock.cc:648] HybridClock initialized: now 1786223403975179 us; error 27 us; skew 500 ppm
Aug 08 21:10:03 dist-test-slave-chsk krb5kdc[27143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:03.977694 27158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:03.978163 27158 webserver.cc:460] Webserver started at http://127.0.0.1:36937/ using document root <none> and password file <none>
I20260808 21:10:03.978400 27158 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:03.978447 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:03.978614 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:03.979728 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "18ab322649c64b7d80e652e281385e80"
format_stamp: "Formatted at 2026-08-08 21:10:03 on dist-test-slave-chsk"
server_key: "28ad102f338a2ad5ac9b18e1beae1ac8"
server_key_iv: "58290d1df2c1e3d8bfdd84a79f7f2340"
server_key_version: "encryptionkey@0"
I20260808 21:10:03.980093 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "18ab322649c64b7d80e652e281385e80"
format_stamp: "Formatted at 2026-08-08 21:10:03 on dist-test-slave-chsk"
server_key: "28ad102f338a2ad5ac9b18e1beae1ac8"
server_key_iv: "58290d1df2c1e3d8bfdd84a79f7f2340"
server_key_version: "encryptionkey@0"
I20260808 21:10:03.981459 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:03.982208 27174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:03.982389 27158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:03.982482 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "18ab322649c64b7d80e652e281385e80"
format_stamp: "Formatted at 2026-08-08 21:10:03 on dist-test-slave-chsk"
server_key: "28ad102f338a2ad5ac9b18e1beae1ac8"
server_key_iv: "58290d1df2c1e3d8bfdd84a79f7f2340"
server_key_version: "encryptionkey@0"
I20260808 21:10:03.982553 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:03.996457 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:03.998337 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:03.998502 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:04.002663 27158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38295
I20260808 21:10:04.002722 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38295 every 8 connection(s)
I20260808 21:10:04.003072 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:04.004052 27232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:04.006330 27232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80: Bootstrap starting.
I20260808 21:10:04.006943 27232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:04.007261 27232 log.cc:824] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:04.007771 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27158
I20260808 21:10:04.007929 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:04.008111 27232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80: No bootstrap required, opened a new log
I20260808 21:10:04.008143 4805 external_mini_cluster.cc:1428] Setting key 02873a0519a000ff86b132cb948430e2
I20260808 21:10:04.010886 27232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260808 21:10:04.011104 27232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:04.011178 27232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ab322649c64b7d80e652e281385e80, State: Initialized, Role: FOLLOWER
I20260808 21:10:04.011313 27232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260808 21:10:04.011395 27232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:04.011435 27232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:04.011480 27232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:04.012167 27232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260808 21:10:04.012295 27232 leader_election.cc:302] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ab322649c64b7d80e652e281385e80; no voters:
I20260808 21:10:04.012495 27232 leader_election.cc:288] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:04.012537 27237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:04.012758 27237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [term 1 LEADER]: Becoming Leader. State: Replica: 18ab322649c64b7d80e652e281385e80, State: Running, Role: LEADER
I20260808 21:10:04.012851 27232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:04.012882 27237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } }
I20260808 21:10:04.013270 27237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ab322649c64b7d80e652e281385e80. Latest consensus state: current_term: 1 leader_uuid: "18ab322649c64b7d80e652e281385e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260808 21:10:04.013401 27237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:04.013672 27245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:04.014209 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:04.008786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51099 (local address 127.0.0.1:38295)
0808 21:10:04.008971 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:04.008979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:04.009003 (+ 24us) server_negotiation.cc:407] Connection header received
0808 21:10:04.009545 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:04.009552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:04.009600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:04.009717 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:04.009988 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:04.010855 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:04.011970 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:04.012428 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:04.012543 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:04.013306 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:04.013310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:04.013310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:04.013530 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:04.013570 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:04.013585 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:04.013661 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:04.013943 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 21:10:04.014015 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260808 21:10:04.014487 27238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18ab322649c64b7d80e652e281385e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ab322649c64b7d80e652e281385e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38295 } } }
I20260808 21:10:04.014623 27245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:04.014803 27238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:04.015133 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:04.016794 27245 catalog_manager.cc:1347] Generated new cluster ID: 2a81c2d5d4ad418db16e6815f7bcb6ea
I20260808 21:10:04.016855 27245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:04.020153 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:04.016743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59381 (local address 127.0.0.1:38295)
0808 21:10:04.016776 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:04.016777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:04.016890 (+ 113us) server_negotiation.cc:407] Connection header received
0808 21:10:04.016982 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:04.016983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:04.016993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:04.017022 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:04.017193 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:04.017902 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:04.018864 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:04.019170 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:04.019204 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:04.019934 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:04.019935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:04.019935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:04.019939 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:04.019952 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:04.019956 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:04.019981 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:04.020067 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 21:10:04.020085 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:10:04.024745 27245 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:04.025408 27245 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:04.028637 27245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ab322649c64b7d80e652e281385e80: Generated new TSK 0
I20260808 21:10:04.028822 27245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:04.033039 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27158
2026-08-08T21:10:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:04 dist-test-slave-chsk krb5kdc[27266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:04 dist-test-slave-chsk krb5kdc[27266](info): set up 2 sockets
Aug 08 21:10:04 dist-test-slave-chsk krb5kdc[27266](info): commencing operation
krb5kdc: starting...
W20260808 21:10:05.103641 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:05 dist-test-slave-chsk krb5kdc[27266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:05.223568 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:05.303658 27279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:05.303855 27279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:05.303879 27279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:05.305064 27279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:05.305107 27279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:05.305122 27279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:05.305160 27279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:05.305194 27279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:05.306669 27279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:05.306913 27279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:05.307166 27279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:10:05.311949 27279 server_base.cc:1034] running on GCE node
W20260808 21:10:05.311932 27288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:05.311913 27286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:05.311932 27285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:05.312358 27279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:05.312644 27279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:05.313854 27279 hybrid_clock.cc:648] HybridClock initialized: now 1786223405313837 us; error 26 us; skew 500 ppm
Aug 08 21:10:05 dist-test-slave-chsk krb5kdc[27266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:05.321465 27279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:05.321969 27279 webserver.cc:460] Webserver started at http://127.0.0.1:46569/ using document root <none> and password file <none>
I20260808 21:10:05.322218 27279 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:05.322288 27279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:05.322444 27279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:05.323635 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "baad109c54cb48a4a146f5e4e0ceb370"
format_stamp: "Formatted at 2026-08-08 21:10:05 on dist-test-slave-chsk"
server_key: "e4d3d0bae68e2efcf29fbace1bee315c"
server_key_iv: "fe69b91179f0979b79f5964df409ea26"
server_key_version: "encryptionkey@0"
I20260808 21:10:05.324009 27279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "baad109c54cb48a4a146f5e4e0ceb370"
format_stamp: "Formatted at 2026-08-08 21:10:05 on dist-test-slave-chsk"
server_key: "e4d3d0bae68e2efcf29fbace1bee315c"
server_key_iv: "fe69b91179f0979b79f5964df409ea26"
server_key_version: "encryptionkey@0"
I20260808 21:10:05.325424 27279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:05.326278 27295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:05.326541 27279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:10:05.326675 27279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "baad109c54cb48a4a146f5e4e0ceb370"
format_stamp: "Formatted at 2026-08-08 21:10:05 on dist-test-slave-chsk"
server_key: "e4d3d0bae68e2efcf29fbace1bee315c"
server_key_iv: "fe69b91179f0979b79f5964df409ea26"
server_key_version: "encryptionkey@0"
I20260808 21:10:05.326774 27279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:05.330102 27279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:05.331990 27279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:05.332144 27279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:05.337076 27279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43147
I20260808 21:10:05.337134 27353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43147 every 8 connection(s)
I20260808 21:10:05.337463 27279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:05.338068 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27279
I20260808 21:10:05.338213 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:05.338434 4805 external_mini_cluster.cc:1428] Setting key cef9fa90cca404d6d8b590e431c41b76
I20260808 21:10:05.338660 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:05.341040 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:05.339198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36511 (local address 127.0.0.1:43147)
0808 21:10:05.339383 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:05.339390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:05.339415 (+ 25us) server_negotiation.cc:407] Connection header received
0808 21:10:05.339793 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:05.339799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:05.339822 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:05.339963 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:05.340629 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:05.340633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:05.340633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:05.340637 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:05.340684 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:05.340694 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:05.340772 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:05.340839 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 21:10:05.340879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260808 21:10:05.341913 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370: Bootstrap starting.
I20260808 21:10:05.342027 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:05.342561 27354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:05.342985 27354 log.cc:824] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:05.344069 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370: No bootstrap required, opened a new log
I20260808 21:10:05.344549 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:05.343474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35189 (local address 127.0.0.1:43147)
0808 21:10:05.343509 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:05.343510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:05.343638 (+ 128us) server_negotiation.cc:407] Connection header received
0808 21:10:05.343701 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:05.343702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:05.343706 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:05.343754 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:05.344356 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:05.344357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:05.344357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:05.344361 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:05.344373 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:05.344377 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:05.344422 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:05.344475 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:10:05.344488 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260808 21:10:05.346079 27354 raft_consensus.cc:348] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } }
I20260808 21:10:05.346271 27354 raft_consensus.cc:374] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:05.346330 27354 raft_consensus.cc:729] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baad109c54cb48a4a146f5e4e0ceb370, State: Initialized, Role: FOLLOWER
I20260808 21:10:05.346450 27354 consensus_queue.cc:260] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } }
I20260808 21:10:05.346547 27354 raft_consensus.cc:388] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:05.346590 27354 raft_consensus.cc:482] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:05.346635 27354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:05.347105 27354 raft_consensus.cc:504] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } }
I20260808 21:10:05.347249 27354 leader_election.cc:302] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baad109c54cb48a4a146f5e4e0ceb370; no voters:
I20260808 21:10:05.347406 27354 leader_election.cc:288] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:05.347460 27364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:05.347610 27364 raft_consensus.cc:686] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [term 1 LEADER]: Becoming Leader. State: Replica: baad109c54cb48a4a146f5e4e0ceb370, State: Running, Role: LEADER
I20260808 21:10:05.347694 27354 sys_catalog.cc:564] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:05.347757 27364 consensus_queue.cc:237] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } }
I20260808 21:10:05.348170 27367 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baad109c54cb48a4a146f5e4e0ceb370. Latest consensus state: current_term: 1 leader_uuid: "baad109c54cb48a4a146f5e4e0ceb370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } } }
I20260808 21:10:05.348188 27365 sys_catalog.cc:455] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baad109c54cb48a4a146f5e4e0ceb370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baad109c54cb48a4a146f5e4e0ceb370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43147 } } }
I20260808 21:10:05.348341 27365 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:05.348327 27367 sys_catalog.cc:458] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:05.348747 27374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:05.349248 27374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:05.350798 27374 catalog_manager.cc:1347] Generated new cluster ID: 814771b12daa4323acf81e8696e31300
I20260808 21:10:05.350852 27374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:05.355136 27374 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:05.355329 27374 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:05.363293 27374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baad109c54cb48a4a146f5e4e0ceb370: Generated new TSK 0
I20260808 21:10:05.363461 27374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:05.366796 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27279
2026-08-08T21:10:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:05 dist-test-slave-chsk krb5kdc[27387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:05 dist-test-slave-chsk krb5kdc[27387](info): set up 2 sockets
Aug 08 21:10:05 dist-test-slave-chsk krb5kdc[27387](info): commencing operation
krb5kdc: starting...
W20260808 21:10:06.440670 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:06.562565 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:06.642998 27401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:06.643265 27401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:06.643306 27401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:06.644507 27401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:06.644574 27401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:06.644601 27401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:06.644665 27401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:06.644723 27401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:06.646234 27401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:06.646486 27401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:06.646723 27401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:06.651470 27409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:06.651475 27406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:06.651479 27407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:06.651685 27401 server_base.cc:1034] running on GCE node
I20260808 21:10:06.651988 27401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:06.652314 27401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:06.653482 27401 hybrid_clock.cc:648] HybridClock initialized: now 1786223406653458 us; error 42 us; skew 500 ppm
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:06.656319 27401 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:06.656838 27401 webserver.cc:460] Webserver started at http://127.0.0.1:37187/ using document root <none> and password file <none>
I20260808 21:10:06.657068 27401 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:06.657135 27401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:06.657271 27401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:06.658298 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "cf77b62d0bf54005a8e05ac6ba754da9"
format_stamp: "Formatted at 2026-08-08 21:10:06 on dist-test-slave-chsk"
server_key: "9823525895ef290bfffb6c0b29e67612"
server_key_iv: "10726963d8a2a4c717fa1d47747814c4"
server_key_version: "encryptionkey@0"
I20260808 21:10:06.658672 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "cf77b62d0bf54005a8e05ac6ba754da9"
format_stamp: "Formatted at 2026-08-08 21:10:06 on dist-test-slave-chsk"
server_key: "9823525895ef290bfffb6c0b29e67612"
server_key_iv: "10726963d8a2a4c717fa1d47747814c4"
server_key_version: "encryptionkey@0"
I20260808 21:10:06.660074 27401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:10:06.660799 27416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:06.660971 27401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:10:06.661075 27401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "cf77b62d0bf54005a8e05ac6ba754da9"
format_stamp: "Formatted at 2026-08-08 21:10:06 on dist-test-slave-chsk"
server_key: "9823525895ef290bfffb6c0b29e67612"
server_key_iv: "10726963d8a2a4c717fa1d47747814c4"
server_key_version: "encryptionkey@0"
I20260808 21:10:06.661154 27401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:06.672834 27401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:06.675086 27401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:06.675287 27401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:06.679369 27401 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39917
I20260808 21:10:06.679427 27476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39917 every 8 connection(s)
I20260808 21:10:06.679787 27401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:06.680740 27477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:06.682999 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9: Bootstrap starting.
I20260808 21:10:06.683619 27477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:06.683898 27477 log.cc:824] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:06.684607 27477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9: No bootstrap required, opened a new log
I20260808 21:10:06.686182 27477 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260808 21:10:06.686309 27477 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:06.686331 27477 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf77b62d0bf54005a8e05ac6ba754da9, State: Initialized, Role: FOLLOWER
I20260808 21:10:06.686434 27477 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260808 21:10:06.686477 27477 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:06.686545 27477 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:06.686592 27477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:06.687115 27477 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260808 21:10:06.687304 27477 leader_election.cc:302] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf77b62d0bf54005a8e05ac6ba754da9; no voters:
I20260808 21:10:06.687366 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27401
I20260808 21:10:06.687497 27477 leader_election.cc:288] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:06.687505 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:06.687526 27480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:06.687644 27480 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [term 1 LEADER]: Becoming Leader. State: Replica: cf77b62d0bf54005a8e05ac6ba754da9, State: Running, Role: LEADER
I20260808 21:10:06.687762 4805 external_mini_cluster.cc:1428] Setting key b2097872bfc50321d5d1462103cc5c38
I20260808 21:10:06.687773 27477 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:06.687788 27480 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260808 21:10:06.688141 27480 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf77b62d0bf54005a8e05ac6ba754da9. Latest consensus state: current_term: 1 leader_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260808 21:10:06.688280 27480 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:06.688316 27481 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf77b62d0bf54005a8e05ac6ba754da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260808 21:10:06.688467 27481 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:06.688971 27489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:06.689635 27489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:06.691161 27489 catalog_manager.cc:1347] Generated new cluster ID: 2429168d472a4bdc8f594b4e40385746
I20260808 21:10:06.691218 27489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 21:10:06.698356 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:06.688433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:51723 (local address 127.0.0.1:39917)
0808 21:10:06.688627 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:06.688638 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:06.688704 (+ 66us) server_negotiation.cc:407] Connection header received
0808 21:10:06.688871 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:06.688879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:06.688939 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:06.689089 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:06.690062 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:06.691381 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:06.692131 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:06.692547 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:06.692669 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:06.694958 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:06.694961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:06.694962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:06.694973 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:06.696731 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:06.697131 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:06.697135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:06.697135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:06.697242 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:06.697479 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:06.697480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:06.697480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:06.697719 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:06.697899 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:06.697985 (+ 86us) server_negotiation.cc:299] Negotiation successful
0808 21:10:06.698090 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260808 21:10:06.699375 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:06.706820 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:06.700778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:54025 (local address 127.0.0.1:39917)
0808 21:10:06.700816 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:06.700818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:06.700941 (+ 123us) server_negotiation.cc:407] Connection header received
0808 21:10:06.701011 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:06.701012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:06.701021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:06.701065 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:06.701574 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:06.702090 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:06.702759 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:06.702982 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:06.703058 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:06.704239 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:06.704240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:06.704240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:06.704247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:06.705623 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:06.706000 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:06.706001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:06.706002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:06.706061 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:06.706291 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:06.706292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:06.706292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:06.706402 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:06.706511 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:06.706632 (+ 121us) server_negotiation.cc:299] Negotiation successful
0808 21:10:06.706703 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:10:06.712566 27489 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:06.713055 27489 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:06.716166 27489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf77b62d0bf54005a8e05ac6ba754da9: Generated new TSK 0
I20260808 21:10:06.716351 27489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:06.718743 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27401
W20260808 21:10:06.725579 27384 connection.cc:535] client connection to 127.0.0.1:39917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39917): Connection reset by peer (error 104)
2026-08-08T21:10:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27511](info): set up 2 sockets
Aug 08 21:10:06 dist-test-slave-chsk krb5kdc[27511](info): commencing operation
krb5kdc: starting...
W20260808 21:10:07.799646 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:07 dist-test-slave-chsk krb5kdc[27511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:07.925616 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:08.005703 27524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:08.005950 27524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:08.005990 27524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:08.007121 27524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:08.007218 27524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:08.007243 27524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:08.007265 27524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:08.007293 27524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:08.008749 27524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:08.009032 27524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:08.009270 27524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:08.014115 27530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:08.014209 27533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:08.014127 27531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:08.014375 27524 server_base.cc:1034] running on GCE node
I20260808 21:10:08.014678 27524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:08.014969 27524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:08.016148 27524 hybrid_clock.cc:648] HybridClock initialized: now 1786223408016120 us; error 54 us; skew 500 ppm
Aug 08 21:10:08 dist-test-slave-chsk krb5kdc[27511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:08.018603 27524 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:08.019100 27524 webserver.cc:460] Webserver started at http://127.0.0.1:41957/ using document root <none> and password file <none>
I20260808 21:10:08.019359 27524 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:08.019430 27524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:08.019588 27524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:08.020890 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "6665792f7fd946d6899ecaaf3ce6cf07"
format_stamp: "Formatted at 2026-08-08 21:10:08 on dist-test-slave-chsk"
server_key: "f56919f37fd68110edfdf80ca299dcc7"
server_key_iv: "e9bc31a00b66fe57065a1426c13a9ade"
server_key_version: "encryptionkey@0"
I20260808 21:10:08.021353 27524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "6665792f7fd946d6899ecaaf3ce6cf07"
format_stamp: "Formatted at 2026-08-08 21:10:08 on dist-test-slave-chsk"
server_key: "f56919f37fd68110edfdf80ca299dcc7"
server_key_iv: "e9bc31a00b66fe57065a1426c13a9ade"
server_key_version: "encryptionkey@0"
I20260808 21:10:08.022774 27524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:10:08.023625 27541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:08.023813 27524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:10:08.023922 27524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "6665792f7fd946d6899ecaaf3ce6cf07"
format_stamp: "Formatted at 2026-08-08 21:10:08 on dist-test-slave-chsk"
server_key: "f56919f37fd68110edfdf80ca299dcc7"
server_key_iv: "e9bc31a00b66fe57065a1426c13a9ade"
server_key_version: "encryptionkey@0"
I20260808 21:10:08.023999 27524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:08.037912 27524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:08.039909 27524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:08.040096 27524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:08.044518 27524 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35421
I20260808 21:10:08.044553 27595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35421 every 8 connection(s)
I20260808 21:10:08.044915 27524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:08.045912 27596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:08.048355 27596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07: Bootstrap starting.
I20260808 21:10:08.048950 27596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:08.049255 27596 log.cc:824] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:08.049995 27596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07: No bootstrap required, opened a new log
I20260808 21:10:08.050340 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27524
I20260808 21:10:08.050472 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:08.050699 4805 external_mini_cluster.cc:1428] Setting key df4333d955fcab3ac7d7d22688b3f6ed
I20260808 21:10:08.052582 27596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260808 21:10:08.052865 27596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:08.052928 27596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6665792f7fd946d6899ecaaf3ce6cf07, State: Initialized, Role: FOLLOWER
I20260808 21:10:08.053104 27596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260808 21:10:08.053174 27596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:08.053216 27596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:08.053273 27596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:08.053851 27596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260808 21:10:08.053937 27596 leader_election.cc:302] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6665792f7fd946d6899ecaaf3ce6cf07; no voters:
I20260808 21:10:08.054124 27596 leader_election.cc:288] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:08.054157 27601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:08.054268 27601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [term 1 LEADER]: Becoming Leader. State: Replica: 6665792f7fd946d6899ecaaf3ce6cf07, State: Running, Role: LEADER
I20260808 21:10:08.054409 27596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:08.054426 27601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } }
I20260808 21:10:08.054840 27601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6665792f7fd946d6899ecaaf3ce6cf07. Latest consensus state: current_term: 1 leader_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } } }
I20260808 21:10:08.054849 27602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6665792f7fd946d6899ecaaf3ce6cf07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35421 } } }
I20260808 21:10:08.055022 27602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:08.055269 27601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:08.055657 27610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:08.056264 27610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:08.058039 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:08.051284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:50167 (local address 127.0.0.1:35421)
0808 21:10:08.051516 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:08.051525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:08.051560 (+ 35us) server_negotiation.cc:407] Connection header received
0808 21:10:08.052542 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:08.052552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:08.052634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:08.052783 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:08.053004 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:08.054475 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:08.055653 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:08.056254 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:08.056359 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:08.057266 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:08.057270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:08.057271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:08.057518 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:08.057562 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:08.057574 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:08.057666 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:08.057799 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 21:10:08.057828 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":116,"threads_started":1}
I20260808 21:10:08.058077 27610 catalog_manager.cc:1347] Generated new cluster ID: b2cd8f2c3c464712bc5b74e759cf9c74
I20260808 21:10:08.058127 27610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:08.058992 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:08.064287 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:08.060474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:39641 (local address 127.0.0.1:35421)
0808 21:10:08.060507 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:08.060509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:08.060646 (+ 137us) server_negotiation.cc:407] Connection header received
0808 21:10:08.060678 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:08.060679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:08.060703 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:08.060776 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:08.060905 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:08.061613 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:08.062574 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:08.063110 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:08.063206 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:08.064004 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:08.064005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:08.064005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:08.064010 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:08.064026 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:08.064031 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:08.064099 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:08.064187 (+ 88us) server_negotiation.cc:299] Negotiation successful
0808 21:10:08.064202 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:10:08.064709 27610 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:08.065191 27610 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:08.069911 27610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6665792f7fd946d6899ecaaf3ce6cf07: Generated new TSK 0
I20260808 21:10:08.070083 27610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:08.071887 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27524
W20260808 21:10:08.078460 27508 connection.cc:535] client connection to 127.0.0.1:35421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35421): Connection reset by peer (error 104)
2026-08-08T21:10:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:08 dist-test-slave-chsk krb5kdc[27628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:08 dist-test-slave-chsk krb5kdc[27628](info): set up 2 sockets
Aug 08 21:10:08 dist-test-slave-chsk krb5kdc[27628](info): commencing operation
krb5kdc: starting...
W20260808 21:10:09.147019 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:09 dist-test-slave-chsk krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:09.274204 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:09.355438 27643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:09.355665 27643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:09.355688 27643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:09.356824 27643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:09.356866 27643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:09.356880 27643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:09.356894 27643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:09.356945 27643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:09.358402 27643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:09.358667 27643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:09.358898 27643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:09.363652 27651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:09.363633 27649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:09.363633 27653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:09.363791 27643 server_base.cc:1034] running on GCE node
I20260808 21:10:09.364113 27643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:09.364430 27643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:09.365600 27643 hybrid_clock.cc:648] HybridClock initialized: now 1786223409365587 us; error 31 us; skew 500 ppm
Aug 08 21:10:09 dist-test-slave-chsk krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:09.368232 27643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:09.368705 27643 webserver.cc:460] Webserver started at http://127.0.0.1:41493/ using document root <none> and password file <none>
I20260808 21:10:09.368940 27643 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:09.369009 27643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:09.369170 27643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:09.370208 27643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "7d4d86129b824c22aef17f194176ae08"
format_stamp: "Formatted at 2026-08-08 21:10:09 on dist-test-slave-chsk"
server_key: "de6f7e17b114ca75c457298fdd2ebb5a"
server_key_iv: "6c463d5aee360d548f04aa6516760853"
server_key_version: "encryptionkey@0"
I20260808 21:10:09.370571 27643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "7d4d86129b824c22aef17f194176ae08"
format_stamp: "Formatted at 2026-08-08 21:10:09 on dist-test-slave-chsk"
server_key: "de6f7e17b114ca75c457298fdd2ebb5a"
server_key_iv: "6c463d5aee360d548f04aa6516760853"
server_key_version: "encryptionkey@0"
I20260808 21:10:09.371996 27643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:10:09.372792 27661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:09.372961 27643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:10:09.373070 27643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "7d4d86129b824c22aef17f194176ae08"
format_stamp: "Formatted at 2026-08-08 21:10:09 on dist-test-slave-chsk"
server_key: "de6f7e17b114ca75c457298fdd2ebb5a"
server_key_iv: "6c463d5aee360d548f04aa6516760853"
server_key_version: "encryptionkey@0"
I20260808 21:10:09.373148 27643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:09.376441 27643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:09.378700 27643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:09.378854 27643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:09.383011 27643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40431
I20260808 21:10:09.383057 27715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40431 every 8 connection(s)
I20260808 21:10:09.383414 27643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:09.384338 27716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:09.386535 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08: Bootstrap starting.
I20260808 21:10:09.387199 27716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:09.387501 27716 log.cc:824] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:09.388206 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08: No bootstrap required, opened a new log
I20260808 21:10:09.389243 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27643
I20260808 21:10:09.389369 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:09.389575 4805 external_mini_cluster.cc:1428] Setting key f445543d9b3ee05fee7d03a5f7049170
I20260808 21:10:09.390424 27716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } }
I20260808 21:10:09.390594 27716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:09.390674 27716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4d86129b824c22aef17f194176ae08, State: Initialized, Role: FOLLOWER
I20260808 21:10:09.390847 27716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } }
I20260808 21:10:09.390933 27716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:09.390956 27716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:09.391027 27716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:09.391901 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:09.390120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:43661 (local address 127.0.0.1:40431)
0808 21:10:09.390307 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:09.390315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:09.390351 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:10:09.390443 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:09.390448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:09.390471 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:09.390574 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:09.391388 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:09.391391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:09.391391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:09.391395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:09.391435 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:09.391446 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:09.391513 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:09.391608 (+ 95us) server_negotiation.cc:299] Negotiation successful
0808 21:10:09.391642 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260808 21:10:09.392165 27716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } }
I20260808 21:10:09.392279 27716 leader_election.cc:302] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d4d86129b824c22aef17f194176ae08; no voters:
I20260808 21:10:09.392438 27716 leader_election.cc:288] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:09.392457 27722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:09.392668 27722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4d86129b824c22aef17f194176ae08, State: Running, Role: LEADER
I20260808 21:10:09.392726 27716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:09.392896 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:09.392887 27722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } }
I20260808 21:10:09.393275 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d4d86129b824c22aef17f194176ae08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } } }
I20260808 21:10:09.393370 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:09.394380 27729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:09.394532 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d4d86129b824c22aef17f194176ae08. Latest consensus state: current_term: 1 leader_uuid: "7d4d86129b824c22aef17f194176ae08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d4d86129b824c22aef17f194176ae08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40431 } } }
I20260808 21:10:09.394733 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:09.394973 27729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:09.396543 27729 catalog_manager.cc:1347] Generated new cluster ID: f6baa8660c854a758a1acef16d4a19fa
I20260808 21:10:09.396606 27729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:09.396799 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:09.395487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:41139 (local address 127.0.0.1:40431)
0808 21:10:09.395529 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:09.395531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:09.395688 (+ 157us) server_negotiation.cc:407] Connection header received
0808 21:10:09.395750 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:09.395750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:09.395767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:09.395828 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:09.396483 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:09.396484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:09.396484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:09.396489 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:09.396504 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:09.396508 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:09.396560 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:09.396700 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 21:10:09.396728 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:10:09.406677 27729 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:09.406849 27729 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:09.411513 27729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d4d86129b824c22aef17f194176ae08: Generated new TSK 0
I20260808 21:10:09.411700 27729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:09.415004 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27643
2026-08-08T21:10:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:09 dist-test-slave-chsk krb5kdc[27749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:09 dist-test-slave-chsk krb5kdc[27749](info): set up 2 sockets
Aug 08 21:10:09 dist-test-slave-chsk krb5kdc[27749](info): commencing operation
krb5kdc: starting...
W20260808 21:10:10.492707 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:10.618880 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:10.700331 27763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:10.700568 27763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:10.700608 27763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:10.701728 27763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:10.701795 27763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:10.701820 27763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:10.701848 27763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:10.701880 27763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:10.703414 27763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:10.703691 27763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:10.703917 27763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:10.708958 27769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:10.709004 27771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:10.709009 27768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:10.709048 27763 server_base.cc:1034] running on GCE node
I20260808 21:10:10.709360 27763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:10.709686 27763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:10.710855 27763 hybrid_clock.cc:648] HybridClock initialized: now 1786223410710841 us; error 28 us; skew 500 ppm
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:10.716794 27763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:10.717258 27763 webserver.cc:460] Webserver started at http://127.0.0.1:36443/ using document root <none> and password file <none>
I20260808 21:10:10.717504 27763 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:10.717579 27763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:10.717730 27763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:10.718876 27763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "459e827b24c744b8b5f2838d0bfce38b"
format_stamp: "Formatted at 2026-08-08 21:10:10 on dist-test-slave-chsk"
server_key: "7c48bd8d6292663a8e9bc7ed786ec605"
server_key_iv: "c95440dd2940ca6912e05ed3f6467963"
server_key_version: "encryptionkey@0"
I20260808 21:10:10.719313 27763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "459e827b24c744b8b5f2838d0bfce38b"
format_stamp: "Formatted at 2026-08-08 21:10:10 on dist-test-slave-chsk"
server_key: "7c48bd8d6292663a8e9bc7ed786ec605"
server_key_iv: "c95440dd2940ca6912e05ed3f6467963"
server_key_version: "encryptionkey@0"
I20260808 21:10:10.720762 27763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:10.721545 27779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:10.721747 27763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:10:10.721853 27763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "459e827b24c744b8b5f2838d0bfce38b"
format_stamp: "Formatted at 2026-08-08 21:10:10 on dist-test-slave-chsk"
server_key: "7c48bd8d6292663a8e9bc7ed786ec605"
server_key_iv: "c95440dd2940ca6912e05ed3f6467963"
server_key_version: "encryptionkey@0"
I20260808 21:10:10.721936 27763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:10.737944 27763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:10.740358 27763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:10.740525 27763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:10.744704 27763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35893
I20260808 21:10:10.744760 27834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35893 every 8 connection(s)
I20260808 21:10:10.745114 27763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:10.746050 27836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:10.748394 27836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b: Bootstrap starting.
I20260808 21:10:10.748986 27836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:10.749291 27836 log.cc:824] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:10.749990 27836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b: No bootstrap required, opened a new log
I20260808 21:10:10.751631 27836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260808 21:10:10.751762 27836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:10.751784 27836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459e827b24c744b8b5f2838d0bfce38b, State: Initialized, Role: FOLLOWER
I20260808 21:10:10.751911 27836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260808 21:10:10.751996 27836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:10.752019 27836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:10.752046 27836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:10.752529 27836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260808 21:10:10.752619 27836 leader_election.cc:302] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459e827b24c744b8b5f2838d0bfce38b; no voters:
I20260808 21:10:10.752736 27836 leader_election.cc:288] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:10.752797 27839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:10.752972 27839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [term 1 LEADER]: Becoming Leader. State: Replica: 459e827b24c744b8b5f2838d0bfce38b, State: Running, Role: LEADER
I20260808 21:10:10.753000 27836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:10.753069 27839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } }
I20260808 21:10:10.753448 27841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459e827b24c744b8b5f2838d0bfce38b. Latest consensus state: current_term: 1 leader_uuid: "459e827b24c744b8b5f2838d0bfce38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260808 21:10:10.753539 27841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:10.753582 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459e827b24c744b8b5f2838d0bfce38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459e827b24c744b8b5f2838d0bfce38b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35893 } } }
I20260808 21:10:10.753650 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:10.753787 27847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:10.754350 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27763
I20260808 21:10:10.754475 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:10.754650 4805 external_mini_cluster.cc:1428] Setting key 566297a748b84c10a4b1edc75244ec2f
I20260808 21:10:10.755573 27847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:10.757121 27847 catalog_manager.cc:1347] Generated new cluster ID: 0a98cd0f6e0445d09319d89fed482566
I20260808 21:10:10.757184 27847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 21:10:10.767041 27847 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:10.767757 27847 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:10.770579 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:10.755209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:49337 (local address 127.0.0.1:35893)
0808 21:10:10.759270 (+ 4061us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:10.759279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:10.759316 (+ 37us) server_negotiation.cc:407] Connection header received
0808 21:10:10.759436 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:10.759443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:10.759499 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:10.759646 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:10.760459 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:10.762318 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:10.763433 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:10.764066 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:10.764164 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:10.767208 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:10.767212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:10.767213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:10.767223 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:10.768975 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:10.769387 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:10.769391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:10.769391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:10.769484 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:10.769729 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:10.769730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:10.769730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:10.769952 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:10.770119 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:10.770243 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 21:10:10.770348 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4006,"thread_start_us":72,"threads_started":1}
I20260808 21:10:10.771684 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:10.772600 27847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459e827b24c744b8b5f2838d0bfce38b: Generated new TSK 0
I20260808 21:10:10.772760 27847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:10.782279 27853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:10.775513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:47583 (local address 127.0.0.1:35893)
0808 21:10:10.775547 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:10.775549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:10.775678 (+ 129us) server_negotiation.cc:407] Connection header received
0808 21:10:10.775753 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:10.775754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:10.775764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:10.775838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:10.776480 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:10.777118 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:10.778167 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:10.778362 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:10.778417 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:10.779791 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:10.779793 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:10.779793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:10.779798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:10.781169 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:10.781571 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:10.781572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:10.781572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:10.781621 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:10.781840 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:10.781841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:10.781841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:10.781933 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:10.782009 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:10.782176 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 21:10:10.782222 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260808 21:10:10.784315 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27763
W20260808 21:10:10.790320 27747 connection.cc:535] client connection to 127.0.0.1:35893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35893): Connection reset by peer (error 104)
2026-08-08T21:10:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27867](info): set up 2 sockets
Aug 08 21:10:10 dist-test-slave-chsk krb5kdc[27867](info): commencing operation
krb5kdc: starting...
W20260808 21:10:11.855724 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:11 dist-test-slave-chsk krb5kdc[27867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:11.988159 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:12.071893 27881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:12.072160 27881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:12.072201 27881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:12.073380 27881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:12.073454 27881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:12.073478 27881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:12.073498 27881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:12.073529 27881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:12.075083 27881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:12.075388 27881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:12.075639 27881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:12.080578 27888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:12.080588 27890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:12.080636 27887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:12.080794 27881 server_base.cc:1034] running on GCE node
I20260808 21:10:12.081127 27881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:12.081439 27881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:12.082608 27881 hybrid_clock.cc:648] HybridClock initialized: now 1786223412082604 us; error 36 us; skew 500 ppm
Aug 08 21:10:12 dist-test-slave-chsk krb5kdc[27867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:12.085220 27881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:12.085723 27881 webserver.cc:460] Webserver started at http://127.0.0.1:43321/ using document root <none> and password file <none>
I20260808 21:10:12.085978 27881 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:12.086048 27881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:12.086201 27881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:12.087360 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b5539ec7efc84e8b8ebc5429775391ab"
format_stamp: "Formatted at 2026-08-08 21:10:12 on dist-test-slave-chsk"
server_key: "8fd928845b532531c2f02090b12174d9"
server_key_iv: "c737ef81397d2e39b6d020f9f3ca2349"
server_key_version: "encryptionkey@0"
I20260808 21:10:12.087782 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b5539ec7efc84e8b8ebc5429775391ab"
format_stamp: "Formatted at 2026-08-08 21:10:12 on dist-test-slave-chsk"
server_key: "8fd928845b532531c2f02090b12174d9"
server_key_iv: "c737ef81397d2e39b6d020f9f3ca2349"
server_key_version: "encryptionkey@0"
I20260808 21:10:12.089152 27881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:12.089902 27898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:12.090123 27881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:10:12.090247 27881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b5539ec7efc84e8b8ebc5429775391ab"
format_stamp: "Formatted at 2026-08-08 21:10:12 on dist-test-slave-chsk"
server_key: "8fd928845b532531c2f02090b12174d9"
server_key_iv: "c737ef81397d2e39b6d020f9f3ca2349"
server_key_version: "encryptionkey@0"
I20260808 21:10:12.090342 27881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:12.105189 27881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:12.107275 27881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:12.107445 27881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:12.111900 27881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40939
I20260808 21:10:12.111958 27951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40939 every 8 connection(s)
I20260808 21:10:12.112351 27881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:12.113241 27952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:12.114676 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 27881
I20260808 21:10:12.114779 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:12.115020 4805 external_mini_cluster.cc:1428] Setting key a5f302ae71790f1be8da0aba9b0b5ef3
I20260808 21:10:12.116189 27952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: Bootstrap starting.
I20260808 21:10:12.116972 27952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:12.117295 27952 log.cc:824] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:12.118088 27952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: No bootstrap required, opened a new log
I20260808 21:10:12.120426 27952 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260808 21:10:12.120647 27952 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:12.120693 27952 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5539ec7efc84e8b8ebc5429775391ab, State: Initialized, Role: FOLLOWER
I20260808 21:10:12.120831 27952 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260808 21:10:12.120927 27952 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:12.120962 27952 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:12.121062 27952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:12.121311 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:12.115634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:34539 (local address 127.0.0.1:40939)
0808 21:10:12.115831 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:12.115841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:12.115875 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:10:12.116419 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:12.116427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:12.116493 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:12.116658 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:12.116879 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:12.118263 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:12.119048 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:12.119704 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:12.119852 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:12.120531 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:12.120536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:12.120536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:12.120750 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:12.120792 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:12.120807 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:12.120898 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:12.121049 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 21:10:12.121084 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260808 21:10:12.121750 27952 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260808 21:10:12.121870 27952 leader_election.cc:302] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5539ec7efc84e8b8ebc5429775391ab; no voters:
I20260808 21:10:12.122076 27952 leader_election.cc:288] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:12.122119 27957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:12.122393 27952 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:12.122468 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:12.122406 27957 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [term 1 LEADER]: Becoming Leader. State: Replica: b5539ec7efc84e8b8ebc5429775391ab, State: Running, Role: LEADER
I20260808 21:10:12.122570 27957 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } }
I20260808 21:10:12.123845 27959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5539ec7efc84e8b8ebc5429775391ab. Latest consensus state: current_term: 1 leader_uuid: "b5539ec7efc84e8b8ebc5429775391ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260808 21:10:12.124167 27959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:12.123929 27958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5539ec7efc84e8b8ebc5429775391ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5539ec7efc84e8b8ebc5429775391ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40939 } } }
I20260808 21:10:12.124296 27958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab [sys.catalog]: This master's current role is: LEADER
W20260808 21:10:12.124457 27978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:12.124521 27978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:12.127313 27981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:12.127535 27981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:12.129316 27981 catalog_manager.cc:1347] Generated new cluster ID: e78e8b41621c467091048356eb78ae92
I20260808 21:10:12.129382 27981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:12.130949 27955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:12.124204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:53785 (local address 127.0.0.1:40939)
0808 21:10:12.124242 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:12.124244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:12.127680 (+ 3436us) server_negotiation.cc:407] Connection header received
0808 21:10:12.127762 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:12.127763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:12.127774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:12.127852 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:12.128011 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:12.128691 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:12.129671 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:12.129898 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:12.129975 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:12.130703 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:12.130705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:12.130705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:12.130710 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:12.130725 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:12.130729 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:12.130783 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:12.130844 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 21:10:12.130853 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:10:12.156385 27981 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:12.156984 27981 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:12.165400 27981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5539ec7efc84e8b8ebc5429775391ab: Generated new TSK 0
I20260808 21:10:12.165623 27981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:12.172164 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 27881
W20260808 21:10:12.180070 27865 connection.cc:535] client connection to 127.0.0.1:40939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40939): Connection reset by peer (error 104)
2026-08-08T21:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:12 dist-test-slave-chsk krb5kdc[27985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:12 dist-test-slave-chsk krb5kdc[27985](info): set up 2 sockets
Aug 08 21:10:12 dist-test-slave-chsk krb5kdc[27985](info): commencing operation
krb5kdc: starting...
W20260808 21:10:13.260215 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:13 dist-test-slave-chsk krb5kdc[27985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 21:10:13.406574 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:13.491750 28000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:13.491992 28000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:13.492033 28000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:13.493192 28000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:13.493268 28000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:13.493295 28000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:13.493328 28000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:13.493355 28000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:13.495436 28000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:13.495733 28000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:13.495972 28000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:13.500857 28006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:13.500883 28009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:13.501017 28000 server_base.cc:1034] running on GCE node
W20260808 21:10:13.501150 28005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:13.501421 28000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:13.501760 28000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:13.502931 28000 hybrid_clock.cc:648] HybridClock initialized: now 1786223413502913 us; error 37 us; skew 500 ppm
Aug 08 21:10:13 dist-test-slave-chsk krb5kdc[27985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:13.505549 28000 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 21:10:13.506042 28000 webserver.cc:460] Webserver started at http://127.0.0.1:43699/ using document root <none> and password file <none>
I20260808 21:10:13.506274 28000 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:13.506320 28000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:13.506460 28000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:13.507594 28000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "216bb795ea5646b988f95714bc13c385"
format_stamp: "Formatted at 2026-08-08 21:10:13 on dist-test-slave-chsk"
server_key: "d6b116538a80a937693c0b80a9554788"
server_key_iv: "32cf4c0377b786fc37b27ee4a24354d4"
server_key_version: "encryptionkey@0"
I20260808 21:10:13.507967 28000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "216bb795ea5646b988f95714bc13c385"
format_stamp: "Formatted at 2026-08-08 21:10:13 on dist-test-slave-chsk"
server_key: "d6b116538a80a937693c0b80a9554788"
server_key_iv: "32cf4c0377b786fc37b27ee4a24354d4"
server_key_version: "encryptionkey@0"
I20260808 21:10:13.509467 28000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:13.510522 28017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:13.510705 28000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260808 21:10:13.510812 28000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "216bb795ea5646b988f95714bc13c385"
format_stamp: "Formatted at 2026-08-08 21:10:13 on dist-test-slave-chsk"
server_key: "d6b116538a80a937693c0b80a9554788"
server_key_iv: "32cf4c0377b786fc37b27ee4a24354d4"
server_key_version: "encryptionkey@0"
I20260808 21:10:13.510900 28000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:13.514434 28000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:13.516970 28000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:13.517135 28000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:13.521406 28000 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33975
I20260808 21:10:13.521440 28071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33975 every 8 connection(s)
I20260808 21:10:13.521801 28000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:13.522328 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28000
I20260808 21:10:13.522472 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:13.522745 4805 external_mini_cluster.cc:1428] Setting key fc9b3c79a0aa831d431621aa837f6da2
I20260808 21:10:13.523115 28072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:13.525290 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:13.523296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:43645 (local address 127.0.0.1:33975)
0808 21:10:13.523503 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:13.523514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:13.523561 (+ 47us) server_negotiation.cc:407] Connection header received
0808 21:10:13.524002 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:13.524007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:13.524034 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:13.524187 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:13.524873 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:13.524876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:13.524877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:13.524881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:13.524924 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:13.524934 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:13.525005 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:13.525073 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 21:10:13.525118 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260808 21:10:13.526350 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:13.526737 28072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385: Bootstrap starting.
I20260808 21:10:13.527498 28072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:13.527856 28072 log.cc:824] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:13.528661 28072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385: No bootstrap required, opened a new log
I20260808 21:10:13.529043 28075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:13.527876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:56817 (local address 127.0.0.1:33975)
0808 21:10:13.527920 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:13.527922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:13.528020 (+ 98us) server_negotiation.cc:407] Connection header received
0808 21:10:13.528126 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:13.528127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:13.528133 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:13.528190 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:13.528836 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:13.528837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:13.528837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:13.528842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:13.528857 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:13.528861 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:13.528907 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:13.528960 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:10:13.528975 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 21:10:13.530684 28072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260808 21:10:13.530831 28072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:13.530855 28072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216bb795ea5646b988f95714bc13c385, State: Initialized, Role: FOLLOWER
I20260808 21:10:13.530970 28072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260808 21:10:13.531054 28072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:13.531078 28072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:13.531108 28072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:13.531682 28072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260808 21:10:13.531776 28072 leader_election.cc:302] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 216bb795ea5646b988f95714bc13c385; no voters:
I20260808 21:10:13.531904 28072 leader_election.cc:288] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:13.531949 28083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:13.532128 28083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [term 1 LEADER]: Becoming Leader. State: Replica: 216bb795ea5646b988f95714bc13c385, State: Running, Role: LEADER
I20260808 21:10:13.532182 28072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:13.532295 28083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260808 21:10:13.532673 28084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "216bb795ea5646b988f95714bc13c385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260808 21:10:13.532701 28085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 216bb795ea5646b988f95714bc13c385. Latest consensus state: current_term: 1 leader_uuid: "216bb795ea5646b988f95714bc13c385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216bb795ea5646b988f95714bc13c385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260808 21:10:13.532780 28084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:13.532788 28085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:13.533023 28092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:13.533571 28092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:13.536216 28092 catalog_manager.cc:1347] Generated new cluster ID: 5a1813a4403b44c2850a12a6bc12614b
I20260808 21:10:13.536288 28092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:13.540295 28092 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:13.540457 28092 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:13.545909 28092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 216bb795ea5646b988f95714bc13c385: Generated new TSK 0
I20260808 21:10:13.546090 28092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:13.550956 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28000
2026-08-08T21:10:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:13 dist-test-slave-chsk krb5kdc[28107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:13 dist-test-slave-chsk krb5kdc[28107](info): set up 2 sockets
Aug 08 21:10:13 dist-test-slave-chsk krb5kdc[28107](info): commencing operation
krb5kdc: starting...
W20260808 21:10:14.626605 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:14.746889 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44083
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:44083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:14.827859 28120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:14.828092 28120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:14.828114 28120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:14.829264 28120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:14.829308 28120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:14.829321 28120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:14.829346 28120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:14.829358 28120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:14.830909 28120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:44083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:14.831216 28120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:14.831440 28120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:14.836217 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:14.836226 28125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:14.836306 28127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:14.836524 28120 server_base.cc:1034] running on GCE node
I20260808 21:10:14.836766 28120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:14.837052 28120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:14.838207 28120 hybrid_clock.cc:648] HybridClock initialized: now 1786223414838200 us; error 40 us; skew 500 ppm
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:14.840670 28120 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:14.841044 28120 webserver.cc:460] Webserver started at http://127.4.177.126:46367/ using document root <none> and password file <none>
I20260808 21:10:14.841233 28120 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:14.841276 28120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:14.841403 28120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:14.842419 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "71c8e683d79a4c668dd3cd21e2b17274"
format_stamp: "Formatted at 2026-08-08 21:10:14 on dist-test-slave-chsk"
server_key: "249ee2c78f2829672cd6f3b4e6047283"
server_key_iv: "7eaa6a141df50b1ecb012c47d4b183b3"
server_key_version: "encryptionkey@0"
I20260808 21:10:14.842722 28120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "71c8e683d79a4c668dd3cd21e2b17274"
format_stamp: "Formatted at 2026-08-08 21:10:14 on dist-test-slave-chsk"
server_key: "249ee2c78f2829672cd6f3b4e6047283"
server_key_iv: "7eaa6a141df50b1ecb012c47d4b183b3"
server_key_version: "encryptionkey@0"
I20260808 21:10:14.844178 28120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260808 21:10:14.844900 28137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:14.845083 28120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:14.845192 28120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "71c8e683d79a4c668dd3cd21e2b17274"
format_stamp: "Formatted at 2026-08-08 21:10:14 on dist-test-slave-chsk"
server_key: "249ee2c78f2829672cd6f3b4e6047283"
server_key_iv: "7eaa6a141df50b1ecb012c47d4b183b3"
server_key_version: "encryptionkey@0"
I20260808 21:10:14.845273 28120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:14.856163 28120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:14.858180 28120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:14.858340 28120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:14.862527 28120 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:44083
I20260808 21:10:14.862556 28193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:44083 every 8 connection(s)
I20260808 21:10:14.862905 28120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:14.863816 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:14.866117 28195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: Bootstrap starting.
I20260808 21:10:14.866686 28195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:14.866986 28195 log.cc:824] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:14.867667 28195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: No bootstrap required, opened a new log
I20260808 21:10:14.869225 28195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } }
I20260808 21:10:14.869357 28195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:14.869379 28195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c8e683d79a4c668dd3cd21e2b17274, State: Initialized, Role: FOLLOWER
I20260808 21:10:14.869478 28195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } }
I20260808 21:10:14.869561 28195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:14.869590 28195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:14.869660 28195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:14.870509 28195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } }
I20260808 21:10:14.870623 28195 leader_election.cc:302] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c8e683d79a4c668dd3cd21e2b17274; no voters:
I20260808 21:10:14.870810 28195 leader_election.cc:288] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:14.870889 28198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:14.871088 28195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:14.871086 28198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [term 1 LEADER]: Becoming Leader. State: Replica: 71c8e683d79a4c668dd3cd21e2b17274, State: Running, Role: LEADER
I20260808 21:10:14.871326 28198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } }
I20260808 21:10:14.871712 28201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c8e683d79a4c668dd3cd21e2b17274. Latest consensus state: current_term: 1 leader_uuid: "71c8e683d79a4c668dd3cd21e2b17274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } } }
I20260808 21:10:14.871878 28201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:14.871939 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28120
I20260808 21:10:14.872069 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:14.872310 4805 external_mini_cluster.cc:1428] Setting key 0eb4c8eda502034d06fcd99ecc2e58a9
I20260808 21:10:14.871723 28200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c8e683d79a4c668dd3cd21e2b17274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c8e683d79a4c668dd3cd21e2b17274" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44083 } } }
I20260808 21:10:14.872592 28200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274 [sys.catalog]: This master's current role is: LEADER
W20260808 21:10:14.873078 28215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:14.873273 28215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:14.873335 28208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:14.873471 28208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:14.875022 28208 catalog_manager.cc:1347] Generated new cluster ID: 2b878971dea04945b6399adf67f0e8e7
I20260808 21:10:14.875074 28208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:14.884799 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:14.873014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41335 (local address 127.4.177.126:44083)
0808 21:10:14.873190 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:14.873200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:14.873236 (+ 36us) server_negotiation.cc:407] Connection header received
0808 21:10:14.873353 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:14.873360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:14.873418 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:14.874470 (+ 1052us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:14.874944 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:14.876362 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:14.877402 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:14.878052 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:14.878165 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:14.881058 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:14.881062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:14.881063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:14.881074 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:14.883022 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:14.883498 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:14.883501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:14.883502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:14.883613 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:14.883871 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:14.883872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:14.883872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:14.884172 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:14.884337 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:14.884488 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 21:10:14.884583 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260808 21:10:14.885766 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:14.891511 28208 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:14.892040 28208 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:14.894644 28218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:14.887128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43693 (local address 127.4.177.126:44083)
0808 21:10:14.887223 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:14.887225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:14.887467 (+ 242us) server_negotiation.cc:407] Connection header received
0808 21:10:14.887499 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:14.887500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:14.887510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:14.887577 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:14.888135 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:14.888911 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:14.889867 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:14.890206 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:14.890277 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:14.891718 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:14.891719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:14.891720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:14.891726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:14.893434 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:14.893840 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:14.893841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:14.893842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:14.893921 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:14.894142 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:14.894143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:14.894143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:14.894268 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:14.894383 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:14.894494 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 21:10:14.894553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 21:10:14.896186 28208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c8e683d79a4c668dd3cd21e2b17274: Generated new TSK 0
I20260808 21:10:14.896361 28208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:14.900223 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28120
W20260808 21:10:14.906613 28105 connection.cc:535] client connection to 127.4.177.126:44083 recv error: Network error: failed to read from TLS socket (remote: 127.4.177.126:44083): Connection reset by peer (error 104)
2026-08-08T21:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28228](info): set up 2 sockets
Aug 08 21:10:14 dist-test-slave-chsk krb5kdc[28228](info): commencing operation
krb5kdc: starting...
W20260808 21:10:15.979311 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:16 dist-test-slave-chsk krb5kdc[28228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:16.102281 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34827
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:34827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:16.183029 28242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:16.183283 28242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:16.183326 28242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:16.184468 28242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:16.184541 28242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:16.184564 28242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:16.184587 28242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:16.184616 28242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:16.186085 28242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:34827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:34827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:16.186367 28242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:16.186592 28242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:10:16.191459 28242 server_base.cc:1034] running on GCE node
W20260808 21:10:16.191440 28250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:16.191430 28247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:16.191453 28248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:16.191980 28242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:16.192236 28242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:16.193377 28242 hybrid_clock.cc:648] HybridClock initialized: now 1786223416193365 us; error 29 us; skew 500 ppm
Aug 08 21:10:16 dist-test-slave-chsk krb5kdc[28228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:16.200536 28242 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:16.201001 28242 webserver.cc:460] Webserver started at http://127.4.177.126:43587/ using document root <none> and password file <none>
I20260808 21:10:16.201229 28242 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:16.201298 28242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:16.201447 28242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:16.202514 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b314ab188ed0443ea02f2b26a1d3ef65"
format_stamp: "Formatted at 2026-08-08 21:10:16 on dist-test-slave-chsk"
server_key: "935adb438d9dcea0d8573deabe23fda0"
server_key_iv: "d1ef4f97a8a8c7713dd811dcebdcfcc3"
server_key_version: "encryptionkey@0"
I20260808 21:10:16.202878 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b314ab188ed0443ea02f2b26a1d3ef65"
format_stamp: "Formatted at 2026-08-08 21:10:16 on dist-test-slave-chsk"
server_key: "935adb438d9dcea0d8573deabe23fda0"
server_key_iv: "d1ef4f97a8a8c7713dd811dcebdcfcc3"
server_key_version: "encryptionkey@0"
I20260808 21:10:16.204269 28242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:10:16.204985 28258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:16.205157 28242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:16.205265 28242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b314ab188ed0443ea02f2b26a1d3ef65"
format_stamp: "Formatted at 2026-08-08 21:10:16 on dist-test-slave-chsk"
server_key: "935adb438d9dcea0d8573deabe23fda0"
server_key_iv: "d1ef4f97a8a8c7713dd811dcebdcfcc3"
server_key_version: "encryptionkey@0"
I20260808 21:10:16.205349 28242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:16.226261 28242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:16.228257 28242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:16.228427 28242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:16.232658 28242 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:34827
I20260808 21:10:16.232692 28311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:34827 every 8 connection(s)
I20260808 21:10:16.233076 28242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:16.233939 28313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:16.236280 28313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: Bootstrap starting.
I20260808 21:10:16.236822 28313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:16.237157 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28242
I20260808 21:10:16.237164 28313 log.cc:824] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:16.237300 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:16.237530 4805 external_mini_cluster.cc:1428] Setting key b970f169a7b7e48af27d17c09409d78a
I20260808 21:10:16.237955 28313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: No bootstrap required, opened a new log
I20260808 21:10:16.239779 28313 raft_consensus.cc:348] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } }
I20260808 21:10:16.239912 28313 raft_consensus.cc:374] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:16.239934 28313 raft_consensus.cc:729] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b314ab188ed0443ea02f2b26a1d3ef65, State: Initialized, Role: FOLLOWER
I20260808 21:10:16.240062 28313 consensus_queue.cc:260] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } }
I20260808 21:10:16.240160 28313 raft_consensus.cc:388] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:16.240206 28313 raft_consensus.cc:482] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:16.240259 28313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:16.240823 28313 raft_consensus.cc:504] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } }
I20260808 21:10:16.240940 28313 leader_election.cc:302] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b314ab188ed0443ea02f2b26a1d3ef65; no voters:
I20260808 21:10:16.241109 28313 leader_election.cc:288] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:16.241159 28318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:16.241329 28318 raft_consensus.cc:686] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [term 1 LEADER]: Becoming Leader. State: Replica: b314ab188ed0443ea02f2b26a1d3ef65, State: Running, Role: LEADER
I20260808 21:10:16.241362 28313 sys_catalog.cc:564] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:16.241483 28318 consensus_queue.cc:237] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } }
I20260808 21:10:16.242731 28320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b314ab188ed0443ea02f2b26a1d3ef65. Latest consensus state: current_term: 1 leader_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } } }
I20260808 21:10:16.242830 28320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:16.243005 28319 sys_catalog.cc:455] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b314ab188ed0443ea02f2b26a1d3ef65" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 34827 } } }
I20260808 21:10:16.243974 28319 sys_catalog.cc:458] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65 [sys.catalog]: This master's current role is: LEADER
W20260808 21:10:16.243408 28335 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:16.244462 28335 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:16.244921 28336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:16.245169 28336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:16.245280 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:16.238065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37789 (local address 127.4.177.126:34827)
0808 21:10:16.238270 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:16.238280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:16.238314 (+ 34us) server_negotiation.cc:407] Connection header received
0808 21:10:16.239770 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:16.239778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:16.239829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:16.239993 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:16.240240 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:16.241791 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:16.242971 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:16.243624 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:16.243734 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:16.244520 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:16.244525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:16.244525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:16.244743 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:16.244776 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:16.244787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:16.244888 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:16.245019 (+ 131us) server_negotiation.cc:299] Negotiation successful
0808 21:10:16.245050 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260808 21:10:16.246824 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:16.247265 28336 catalog_manager.cc:1347] Generated new cluster ID: 3d5dc6dc69644456937c34ad2030dfe2
I20260808 21:10:16.247328 28336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:16.251896 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:16.248394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58133 (local address 127.4.177.126:34827)
0808 21:10:16.248435 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:16.248436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:16.248577 (+ 141us) server_negotiation.cc:407] Connection header received
0808 21:10:16.248655 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:16.248656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:16.248667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:16.248718 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:16.248869 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:16.249855 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:16.250586 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:16.250827 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:16.250905 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:16.251647 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:16.251649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:16.251649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:16.251654 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:16.251670 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:16.251674 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:16.251729 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:16.251789 (+ 60us) server_negotiation.cc:299] Negotiation successful
0808 21:10:16.251800 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:10:16.256018 28336 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:16.256547 28336 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:16.263383 28336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b314ab188ed0443ea02f2b26a1d3ef65: Generated new TSK 0
I20260808 21:10:16.263561 28336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:16.266297 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28242
W20260808 21:10:16.272625 28225 connection.cc:535] client connection to 127.4.177.126:34827 recv error: Network error: failed to read from TLS socket (remote: 127.4.177.126:34827): Connection reset by peer (error 104)
2026-08-08T21:10:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:16 dist-test-slave-chsk krb5kdc[28346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:16 dist-test-slave-chsk krb5kdc[28346](info): set up 2 sockets
Aug 08 21:10:16 dist-test-slave-chsk krb5kdc[28346](info): commencing operation
krb5kdc: starting...
W20260808 21:10:17.335486 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:17 dist-test-slave-chsk krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:17.455466 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42189
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:17.537026 28359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:17.537240 28359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:17.537261 28359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:17.538421 28359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:17.538465 28359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:17.538479 28359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:17.538492 28359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:17.538504 28359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:17.540148 28359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:42189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:17.540398 28359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:17.540648 28359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:17.545383 28366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:17.545522 28368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:17.545549 28365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:17.545861 28359 server_base.cc:1034] running on GCE node
I20260808 21:10:17.546098 28359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:17.546334 28359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:17.547468 28359 hybrid_clock.cc:648] HybridClock initialized: now 1786223417547456 us; error 29 us; skew 500 ppm
Aug 08 21:10:17 dist-test-slave-chsk krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223417, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:17.549896 28359 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:17.550271 28359 webserver.cc:460] Webserver started at http://127.4.177.126:35245/ using document root <none> and password file <none>
I20260808 21:10:17.550457 28359 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:17.550495 28359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:17.550585 28359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:17.551654 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74"
format_stamp: "Formatted at 2026-08-08 21:10:17 on dist-test-slave-chsk"
server_key: "097e98dae382acf1fa8dcc3ff173d3d6"
server_key_iv: "b639418cdb5efab3b785c2de12b5cb78"
server_key_version: "encryptionkey@0"
I20260808 21:10:17.551973 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74"
format_stamp: "Formatted at 2026-08-08 21:10:17 on dist-test-slave-chsk"
server_key: "097e98dae382acf1fa8dcc3ff173d3d6"
server_key_iv: "b639418cdb5efab3b785c2de12b5cb78"
server_key_version: "encryptionkey@0"
I20260808 21:10:17.553269 28359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:17.553931 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:17.554103 28359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:17.554209 28359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74"
format_stamp: "Formatted at 2026-08-08 21:10:17 on dist-test-slave-chsk"
server_key: "097e98dae382acf1fa8dcc3ff173d3d6"
server_key_iv: "b639418cdb5efab3b785c2de12b5cb78"
server_key_version: "encryptionkey@0"
I20260808 21:10:17.554287 28359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:17.557538 28359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:17.559423 28359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:17.559571 28359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:17.563625 28359 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:42189
I20260808 21:10:17.563673 28430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:42189 every 8 connection(s)
I20260808 21:10:17.564003 28359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:17.564986 28431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:17.567281 28431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74: Bootstrap starting.
I20260808 21:10:17.567842 28431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:17.568136 28431 log.cc:824] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:17.568816 28431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74: No bootstrap required, opened a new log
I20260808 21:10:17.570199 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28359
I20260808 21:10:17.570317 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:17.570526 4805 external_mini_cluster.cc:1428] Setting key 2354b2f0c9a886dbd0a7e615db59f9fc
I20260808 21:10:17.570753 28431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } }
I20260808 21:10:17.570919 28431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:17.570961 28431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568fe3ea9ae8490aaa3a7a111bbb3f74, State: Initialized, Role: FOLLOWER
I20260808 21:10:17.571100 28431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } }
I20260808 21:10:17.571244 28431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:17.571293 28431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:17.571374 28431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:17.572521 28431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } }
I20260808 21:10:17.572634 28431 leader_election.cc:302] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 568fe3ea9ae8490aaa3a7a111bbb3f74; no voters:
I20260808 21:10:17.572695 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:17.571045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45813 (local address 127.4.177.126:42189)
0808 21:10:17.571217 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:17.571227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:17.571291 (+ 64us) server_negotiation.cc:407] Connection header received
0808 21:10:17.571436 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:17.571442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:17.571474 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:17.571573 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:17.572211 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:17.572214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:17.572214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:17.572217 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:17.572256 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:17.572266 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:17.572319 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:17.572408 (+ 89us) server_negotiation.cc:299] Negotiation successful
0808 21:10:17.572437 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":97,"threads_started":1}
I20260808 21:10:17.572810 28431 leader_election.cc:288] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:17.572956 28437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:17.573042 28431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:17.573071 28437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [term 1 LEADER]: Becoming Leader. State: Replica: 568fe3ea9ae8490aaa3a7a111bbb3f74, State: Running, Role: LEADER
I20260808 21:10:17.573144 28437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } }
I20260808 21:10:17.573467 28437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568fe3ea9ae8490aaa3a7a111bbb3f74. Latest consensus state: current_term: 1 leader_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } } }
I20260808 21:10:17.573482 28438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568fe3ea9ae8490aaa3a7a111bbb3f74" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 42189 } } }
I20260808 21:10:17.573568 28437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:17.573590 28438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74 [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:17.573716 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:17.573796 28445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:17.574301 28445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:17.576452 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:17.575509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43525 (local address 127.4.177.126:42189)
0808 21:10:17.575543 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:17.575545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:17.575554 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:10:17.575619 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:17.575620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:17.575625 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:17.575678 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:17.576245 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:17.576246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:17.576246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:17.576250 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:17.576263 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:17.576267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:17.576314 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:17.576367 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:10:17.576393 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:10:17.580240 28445 catalog_manager.cc:1347] Generated new cluster ID: 33d7c696a37348468b616461962bf045
I20260808 21:10:17.580292 28445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:17.606860 28445 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:17.607079 28445 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:17.618511 28445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 568fe3ea9ae8490aaa3a7a111bbb3f74: Generated new TSK 0
I20260808 21:10:17.618729 28445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:17.630192 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28359
2026-08-08T21:10:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:17 dist-test-slave-chsk krb5kdc[28465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:17 dist-test-slave-chsk krb5kdc[28465](info): set up 2 sockets
Aug 08 21:10:17 dist-test-slave-chsk krb5kdc[28465](info): commencing operation
krb5kdc: starting...
W20260808 21:10:18.707497 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:18 dist-test-slave-chsk krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:18.826063 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35671
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:43951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:18.908309 28479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:18.908519 28479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:18.908545 28479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:18.909610 28479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:18.909646 28479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:18.909659 28479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:18.909672 28479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:18.909683 28479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:18.911245 28479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:43951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:35671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:18.911443 28479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:18.911609 28479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 21:10:18.916445 28479 server_base.cc:1034] running on GCE node
W20260808 21:10:18.916370 28488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:18.916368 28484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:18.916569 28486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:18.916896 28479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:18.917198 28479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:18.918370 28479 hybrid_clock.cc:648] HybridClock initialized: now 1786223418918351 us; error 50 us; skew 500 ppm
Aug 08 21:10:18 dist-test-slave-chsk krb5kdc[28465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223418, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:18.920890 28479 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:18.921380 28479 webserver.cc:460] Webserver started at http://127.4.177.126:43675/ using document root <none> and password file <none>
I20260808 21:10:18.921617 28479 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:18.921687 28479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:18.921841 28479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:18.922940 28479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "d8d38eb671bb4316b604571097b4070a"
format_stamp: "Formatted at 2026-08-08 21:10:18 on dist-test-slave-chsk"
server_key: "6a6653ea46550950deb6dae982d13542"
server_key_iv: "62a6f8d83fd975fbf528d6ce279ac6c0"
server_key_version: "encryptionkey@0"
I20260808 21:10:18.923350 28479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "d8d38eb671bb4316b604571097b4070a"
format_stamp: "Formatted at 2026-08-08 21:10:18 on dist-test-slave-chsk"
server_key: "6a6653ea46550950deb6dae982d13542"
server_key_iv: "62a6f8d83fd975fbf528d6ce279ac6c0"
server_key_version: "encryptionkey@0"
I20260808 21:10:18.924674 28479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:18.925419 28495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:18.925596 28479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:10:18.925704 28479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "d8d38eb671bb4316b604571097b4070a"
format_stamp: "Formatted at 2026-08-08 21:10:18 on dist-test-slave-chsk"
server_key: "6a6653ea46550950deb6dae982d13542"
server_key_iv: "62a6f8d83fd975fbf528d6ce279ac6c0"
server_key_version: "encryptionkey@0"
I20260808 21:10:18.925781 28479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:18.952888 28479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:18.954842 28479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:18.955018 28479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:18.959281 28479 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:35671
I20260808 21:10:18.959326 28549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:35671 every 8 connection(s)
I20260808 21:10:18.959672 28479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:18.960500 28550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:18.961411 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28479
I20260808 21:10:18.961529 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:18.961745 4805 external_mini_cluster.cc:1428] Setting key 404c79c06c7f237af49cf0c3a8fb1f68
I20260808 21:10:18.963717 28550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: Bootstrap starting.
I20260808 21:10:18.964325 28550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:18.964629 28550 log.cc:824] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:18.965319 28550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: No bootstrap required, opened a new log
I20260808 21:10:18.967132 28550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } }
I20260808 21:10:18.967351 28550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:18.967399 28550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d38eb671bb4316b604571097b4070a, State: Initialized, Role: FOLLOWER
I20260808 21:10:18.967600 28550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } }
I20260808 21:10:18.967700 28550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:18.967752 28550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:18.967803 28550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 0 FOLLOWER]: Advancing to term 1
Aug 08 21:10:18 dist-test-slave-chsk krb5kdc[28465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:18.968377 28550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } }
I20260808 21:10:18.968502 28550 leader_election.cc:302] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d38eb671bb4316b604571097b4070a; no voters:
I20260808 21:10:18.968676 28550 leader_election.cc:288] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:18.968724 28555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:18.968891 28555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [term 1 LEADER]: Becoming Leader. State: Replica: d8d38eb671bb4316b604571097b4070a, State: Running, Role: LEADER
I20260808 21:10:18.968950 28550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:18.968990 28555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } }
I20260808 21:10:18.969671 28556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8d38eb671bb4316b604571097b4070a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } } }
I20260808 21:10:18.969763 28556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:18.970271 28557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8d38eb671bb4316b604571097b4070a. Latest consensus state: current_term: 1 leader_uuid: "d8d38eb671bb4316b604571097b4070a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d38eb671bb4316b604571097b4070a" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 35671 } } }
I20260808 21:10:18.970366 28557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a [sys.catalog]: This master's current role is: LEADER
W20260808 21:10:18.971060 28570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:18.971129 28570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:18.971206 28566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:18.971395 28566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:18.972944 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:18.962349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57855 (local address 127.4.177.126:35671)
0808 21:10:18.962548 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:18.962558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:18.962596 (+ 38us) server_negotiation.cc:407] Connection header received
0808 21:10:18.962997 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:18.963004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:18.963048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:18.963259 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:18.963924 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:18.964911 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:18.965709 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:18.966112 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:18.966248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:18.968880 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:18.968885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:18.968886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:18.968897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:18.971396 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:18.971803 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:18.971806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:18.971806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:18.971892 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:18.972131 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:18.972132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:18.972132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:18.972376 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:18.972522 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:18.972629 (+ 107us) server_negotiation.cc:299] Negotiation successful
0808 21:10:18.972782 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260808 21:10:18.974002 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:18.976215 28566 catalog_manager.cc:1347] Generated new cluster ID: 4f15d141077f4623a1b41bf70836a335
I20260808 21:10:18.976269 28566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:18.981770 28553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:18.975402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36459 (local address 127.4.177.126:35671)
0808 21:10:18.975439 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:18.975441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:18.975455 (+ 14us) server_negotiation.cc:407] Connection header received
0808 21:10:18.975523 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:18.975524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:18.975532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:18.975584 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:18.976124 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:18.976656 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:18.977347 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:18.977580 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:18.977657 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:18.978833 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:18.978835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:18.978835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:18.978842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:18.980655 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:18.981017 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:18.981018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:18.981019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:18.981079 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:18.981279 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:18.981281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:18.981281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:18.981392 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:18.981495 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:18.981599 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 21:10:18.981675 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:10:18.985244 28566 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:18.985915 28566 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:18.998594 28566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8d38eb671bb4316b604571097b4070a: Generated new TSK 0
I20260808 21:10:18.998827 28566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:19.000859 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28479
2026-08-08T21:10:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:19 dist-test-slave-chsk krb5kdc[28581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:19 dist-test-slave-chsk krb5kdc[28581](info): set up 2 sockets
Aug 08 21:10:19 dist-test-slave-chsk krb5kdc[28581](info): commencing operation
krb5kdc: starting...
W20260808 21:10:20.076516 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:20 dist-test-slave-chsk krb5kdc[28581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:20.205482 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:36937
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:36937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:20.290100 28594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:20.290303 28594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:20.290326 28594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:20.291637 28594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:20.291687 28594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:20.291704 28594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:20.291718 28594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:20.291756 28594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:20.293289 28594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:36937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:36937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:20.293533 28594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:20.293780 28594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:20.298810 28602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:20.298835 28600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:20.298827 28594 server_base.cc:1034] running on GCE node
W20260808 21:10:20.298779 28599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:20.299324 28594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:20.299630 28594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:20.300817 28594 hybrid_clock.cc:648] HybridClock initialized: now 1786223420300805 us; error 30 us; skew 500 ppm
Aug 08 21:10:20 dist-test-slave-chsk krb5kdc[28581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223420, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:20.303464 28594 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:20.303857 28594 webserver.cc:460] Webserver started at http://127.4.177.126:44789/ using document root <none> and password file <none>
I20260808 21:10:20.304040 28594 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:20.304081 28594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:20.304171 28594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:20.305284 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "26cc255e3608459c8c43b42a8b86244b"
format_stamp: "Formatted at 2026-08-08 21:10:20 on dist-test-slave-chsk"
server_key: "c44f73b99964761b671f8db48ba067c3"
server_key_iv: "2906292cf96d74a2a500c165abc0e1dc"
server_key_version: "encryptionkey@0"
I20260808 21:10:20.305604 28594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "26cc255e3608459c8c43b42a8b86244b"
format_stamp: "Formatted at 2026-08-08 21:10:20 on dist-test-slave-chsk"
server_key: "c44f73b99964761b671f8db48ba067c3"
server_key_iv: "2906292cf96d74a2a500c165abc0e1dc"
server_key_version: "encryptionkey@0"
I20260808 21:10:20.306934 28594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260808 21:10:20.307729 28609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:20.307928 28594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260808 21:10:20.308038 28594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "26cc255e3608459c8c43b42a8b86244b"
format_stamp: "Formatted at 2026-08-08 21:10:20 on dist-test-slave-chsk"
server_key: "c44f73b99964761b671f8db48ba067c3"
server_key_iv: "2906292cf96d74a2a500c165abc0e1dc"
server_key_version: "encryptionkey@0"
I20260808 21:10:20.308117 28594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:20.316815 28594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:20.319132 28594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:20.319391 28594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:20.323462 28594 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:36937
I20260808 21:10:20.323501 28665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:36937 every 8 connection(s)
I20260808 21:10:20.323863 28594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:20.324767 28667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:20.327049 28667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b: Bootstrap starting.
I20260808 21:10:20.327675 28667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:20.327911 28667 log.cc:824] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:20.328604 28667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b: No bootstrap required, opened a new log
I20260808 21:10:20.330163 28667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } }
I20260808 21:10:20.330299 28667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:20.330322 28667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cc255e3608459c8c43b42a8b86244b, State: Initialized, Role: FOLLOWER
I20260808 21:10:20.330456 28667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } }
I20260808 21:10:20.330549 28667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:20.330571 28667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:20.330600 28667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:20.331382 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28594
I20260808 21:10:20.331511 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:20.331581 28667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } }
I20260808 21:10:20.331722 4805 external_mini_cluster.cc:1428] Setting key ee655993b34e5c314d35a79ea18a4de9
I20260808 21:10:20.331722 28667 leader_election.cc:302] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26cc255e3608459c8c43b42a8b86244b; no voters:
I20260808 21:10:20.331888 28667 leader_election.cc:288] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:20.331941 28670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:20.332111 28670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [term 1 LEADER]: Becoming Leader. State: Replica: 26cc255e3608459c8c43b42a8b86244b, State: Running, Role: LEADER
I20260808 21:10:20.332180 28667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:20.332227 28670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } }
I20260808 21:10:20.332597 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cc255e3608459c8c43b42a8b86244b. Latest consensus state: current_term: 1 leader_uuid: "26cc255e3608459c8c43b42a8b86244b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } } }
I20260808 21:10:20.332609 28671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26cc255e3608459c8c43b42a8b86244b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cc255e3608459c8c43b42a8b86244b" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 36937 } } }
I20260808 21:10:20.332691 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:20.332700 28671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:20.332891 28680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:20.333454 28680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:20.335117 28680 catalog_manager.cc:1347] Generated new cluster ID: af4c62fbb6ca4aa0ace86ee64df40e9e
I20260808 21:10:20.335204 28680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:20.338110 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:20.332428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54667 (local address 127.4.177.126:36937)
0808 21:10:20.333170 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:20.333181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:20.333225 (+ 44us) server_negotiation.cc:407] Connection header received
0808 21:10:20.333360 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:20.333368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:20.333437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:20.333873 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:20.333976 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:20.335379 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:20.336177 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:20.336611 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:20.336736 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:20.337490 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:20.337494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:20.337494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:20.337675 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:20.337705 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:20.337717 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:20.337787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:20.337860 (+ 73us) server_negotiation.cc:299] Negotiation successful
0808 21:10:20.337924 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":78,"threads_started":1}
I20260808 21:10:20.339041 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:20.343273 28674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:20.340457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37575 (local address 127.4.177.126:36937)
0808 21:10:20.340494 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:20.340496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:20.340571 (+ 75us) server_negotiation.cc:407] Connection header received
0808 21:10:20.340640 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:20.340640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:20.340650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:20.340706 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:20.340854 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:20.341313 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:20.342021 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:20.342240 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:20.342316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:20.342990 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:20.342991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:20.342991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:20.342997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:20.343011 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:20.343015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:20.343063 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:20.343119 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 21:10:20.343169 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:10:20.345114 28680 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:20.345616 28680 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:20.351991 28680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26cc255e3608459c8c43b42a8b86244b: Generated new TSK 0
I20260808 21:10:20.352195 28680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:20.357868 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28594
2026-08-08T21:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:20 dist-test-slave-chsk krb5kdc[28700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:20 dist-test-slave-chsk krb5kdc[28700](info): set up 2 sockets
Aug 08 21:10:20 dist-test-slave-chsk krb5kdc[28700](info): commencing operation
krb5kdc: starting...
W20260808 21:10:21.439626 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:21 dist-test-slave-chsk krb5kdc[28700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:21.564193 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37077
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:21.646201 28713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:21.646394 28713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:21.646416 28713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:21.647488 28713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:21.647523 28713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:21.647537 28713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:21.647550 28713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:21.647562 28713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:21.648929 28713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:21.649178 28713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:21.649426 28713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:21.654134 28718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:21.654131 28721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:21.654170 28719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:21.654381 28713 server_base.cc:1034] running on GCE node
I20260808 21:10:21.654599 28713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:21.654832 28713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:21.655970 28713 hybrid_clock.cc:648] HybridClock initialized: now 1786223421655955 us; error 29 us; skew 500 ppm
Aug 08 21:10:21 dist-test-slave-chsk krb5kdc[28700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223421, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:21.664892 28713 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:21.665351 28713 webserver.cc:460] Webserver started at http://127.4.177.126:45835/ using document root <none> and password file <none>
I20260808 21:10:21.665637 28713 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:21.665685 28713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:21.665859 28713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:21.666898 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "b53f94f94b5d442a9a27d0169c43d83e"
format_stamp: "Formatted at 2026-08-08 21:10:21 on dist-test-slave-chsk"
server_key: "c2cbbe4d5e879358942af18e18396fd9"
server_key_iv: "4c2da4b3c768128d241fd328a66e32a7"
server_key_version: "encryptionkey@0"
I20260808 21:10:21.667322 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "b53f94f94b5d442a9a27d0169c43d83e"
format_stamp: "Formatted at 2026-08-08 21:10:21 on dist-test-slave-chsk"
server_key: "c2cbbe4d5e879358942af18e18396fd9"
server_key_iv: "4c2da4b3c768128d241fd328a66e32a7"
server_key_version: "encryptionkey@0"
I20260808 21:10:21.668650 28713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:21.669418 28728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:21.669600 28713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:21.669679 28713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "b53f94f94b5d442a9a27d0169c43d83e"
format_stamp: "Formatted at 2026-08-08 21:10:21 on dist-test-slave-chsk"
server_key: "c2cbbe4d5e879358942af18e18396fd9"
server_key_iv: "4c2da4b3c768128d241fd328a66e32a7"
server_key_version: "encryptionkey@0"
I20260808 21:10:21.669729 28713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:21.672938 28713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:21.674691 28713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:21.674810 28713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:21.678808 28713 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:37077
I20260808 21:10:21.678882 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:37077 every 8 connection(s)
I20260808 21:10:21.679229 28713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:21.679675 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28713
I20260808 21:10:21.679797 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:21.679984 4805 external_mini_cluster.cc:1428] Setting key e8e1946774adb972be00dba4321345f3
I20260808 21:10:21.680670 28784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:21.682476 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:21.680508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51027 (local address 127.4.177.126:37077)
0808 21:10:21.680716 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:21.680726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:21.680764 (+ 38us) server_negotiation.cc:407] Connection header received
0808 21:10:21.681229 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:21.681235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:21.681259 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:21.681408 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:21.682067 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:21.682071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:21.682071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:21.682075 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:21.682122 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:21.682132 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:21.682204 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:21.682275 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 21:10:21.682315 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":98,"threads_started":1}
I20260808 21:10:21.683465 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:21.683773 28784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e: Bootstrap starting.
I20260808 21:10:21.684365 28784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:21.684744 28784 log.cc:824] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:21.685462 28784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e: No bootstrap required, opened a new log
I20260808 21:10:21.685945 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:21.684821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51887 (local address 127.4.177.126:37077)
0808 21:10:21.684878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:21.684881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:21.685037 (+ 156us) server_negotiation.cc:407] Connection header received
0808 21:10:21.685108 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:21.685109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:21.685114 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:21.685169 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:21.685745 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:21.685746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:21.685747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 21:10:21.685751 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:21.685763 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 21:10:21.685767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:21.685816 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:21.685869 (+ 53us) server_negotiation.cc:299] Negotiation successful
0808 21:10:21.685882 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 21:10:21.687594 28784 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } }
I20260808 21:10:21.687727 28784 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 21:10:21.687752 28784 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53f94f94b5d442a9a27d0169c43d83e, State: Initialized, Role: FOLLOWER
I20260808 21:10:21.687861 28784 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } }
I20260808 21:10:21.687945 28784 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 21:10:21.687968 28784 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 21:10:21.687994 28784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:21.688455 28784 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } }
I20260808 21:10:21.688535 28784 leader_election.cc:302] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53f94f94b5d442a9a27d0169c43d83e; no voters:
I20260808 21:10:21.688748 28784 leader_election.cc:288] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 21:10:21.688799 28794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:21.688989 28784 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:21.689006 28794 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [term 1 LEADER]: Becoming Leader. State: Replica: b53f94f94b5d442a9a27d0169c43d83e, State: Running, Role: LEADER
I20260808 21:10:21.689087 28794 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } }
I20260808 21:10:21.689481 28795 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53f94f94b5d442a9a27d0169c43d83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } } }
I20260808 21:10:21.689502 28796 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53f94f94b5d442a9a27d0169c43d83e. Latest consensus state: current_term: 1 leader_uuid: "b53f94f94b5d442a9a27d0169c43d83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53f94f94b5d442a9a27d0169c43d83e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 37077 } } }
I20260808 21:10:21.689630 28795 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:21.689637 28796 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:21.689884 28803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:21.690531 28803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:21.692004 28803 catalog_manager.cc:1347] Generated new cluster ID: 7f515fb89e3e4c53988cf1851ad972a9
I20260808 21:10:21.692068 28803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:21.712188 28803 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:21.712365 28803 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:21.717159 28803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53f94f94b5d442a9a27d0169c43d83e: Generated new TSK 0
I20260808 21:10:21.717331 28803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:21.724484 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28713
2026-08-08T21:10:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1368 ms)
[----------] 18 tests from AuthTokenIssuingTest (24510 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:21 dist-test-slave-chsk krb5kdc[28817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:21 dist-test-slave-chsk krb5kdc[28817](info): set up 2 sockets
Aug 08 21:10:21 dist-test-slave-chsk krb5kdc[28817](info): commencing operation
krb5kdc: starting...
W20260808 21:10:22.790807 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:22 dist-test-slave-chsk krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:22.911264 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44311
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:22.990998 28832 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:22.991230 28832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:22.991250 28832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:22.991269 28832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:22.992362 28832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:22.992395 28832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:22.992409 28832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:22.992421 28832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:22.992434 28832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:22.993821 28832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:22.994026 28832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:22.994208 28832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:22.998950 28840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:22.999089 28832 server_base.cc:1034] running on GCE node
W20260808 21:10:22.998987 28837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:22.998993 28838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:22.999390 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:22.999689 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:23.000833 28832 hybrid_clock.cc:648] HybridClock initialized: now 1786223423000817 us; error 41 us; skew 500 ppm
Aug 08 21:10:22 dist-test-slave-chsk krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:23.003405 28832 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:23.003880 28832 webserver.cc:460] Webserver started at http://127.4.177.126:44603/ using document root <none> and password file <none>
I20260808 21:10:23.004127 28832 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:23.004175 28832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:23.004341 28832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:23.005393 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "61085ebf460442fea76ab678d023dc0e"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "87950c62c2600adfdd1beb07df42ad7c"
server_key_iv: "0b243adcb066c715bdd5dd79d0215598"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.005838 28832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "61085ebf460442fea76ab678d023dc0e"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "87950c62c2600adfdd1beb07df42ad7c"
server_key_iv: "0b243adcb066c715bdd5dd79d0215598"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.007248 28832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260808 21:10:23.008004 28847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:23.008195 28832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:23.008285 28832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "61085ebf460442fea76ab678d023dc0e"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "87950c62c2600adfdd1beb07df42ad7c"
server_key_iv: "0b243adcb066c715bdd5dd79d0215598"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.008356 28832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:23.037412 28832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:23.039346 28832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:23.039500 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:23.043707 28832 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:44311
I20260808 21:10:23.043797 28901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:44311 every 8 connection(s)
I20260808 21:10:23.044128 28832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:23.045076 28902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:23.045743 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28832
I20260808 21:10:23.045892 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:23.046104 4805 external_mini_cluster.cc:1428] Setting key adbf2648e84a20f5f731c12df5688756
I20260808 21:10:23.046857 28902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.125@KRBTEST.COM; defaulting to no policy
I20260808 21:10:23.058063 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.047602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.4.177.126:44311)
0808 21:10:23.047834 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.047843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.048014 (+ 171us) server_negotiation.cc:407] Connection header received
0808 21:10:23.048711 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.048713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.048731 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.048814 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.049554 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.050937 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.051879 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.052428 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.052547 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.055002 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.055011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.055011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.055019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.056658 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.057001 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.057003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.057004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.057080 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.057278 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.057279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.057279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.057477 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.057539 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.057775 (+ 236us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.057862 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260808 21:10:23.058064 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.047602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44311 (local address 127.0.0.1:50124)
0808 21:10:23.047843 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.047879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.048557 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.048852 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.048855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.049228 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.049436 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.049441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.051052 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.051052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.051794 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.051797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.052548 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.052549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.052618 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.053189 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.053197 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.054904 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.056730 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.056736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.056738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.056929 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.057137 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.057138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.057138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.057222 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.057563 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.057570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.057666 (+ 96us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.057749 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.057914 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":53,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.125@KRBTEST.COM" created.
I20260808 21:10:23.059084 28902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:23.059441 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.059333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56042)
0808 21:10:23.059380 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.059422 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 21:10:23.059597 28848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
I20260808 21:10:23.059970 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.059908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56054)
0808 21:10:23.059932 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.059955 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260808 21:10:23.060087 28902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34521: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
I20260808 21:10:23.074530 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:34521
--webserver_interface=127.4.177.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:10:23.109301 28902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } attempt: 1
I20260808 21:10:23.109753 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.109618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56056)
0808 21:10:23.109685 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.109721 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 21:10:23.110097 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.110003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56072)
0808 21:10:23.110047 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.110072 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260808 21:10:23.110208 28902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34521: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
W20260808 21:10:23.155018 28910 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:23.155269 28910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:23.155299 28910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:23.155342 28910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:23.156432 28910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:23.156498 28910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:23.156522 28910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:23.156561 28910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:23.156585 28910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:23.158068 28910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.177.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:23.158349 28910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:23.158591 28910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:23.163271 28915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:23.163276 28918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:23.163280 28916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:23.163559 28910 server_base.cc:1034] running on GCE node
I20260808 21:10:23.163800 28910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:23.164084 28910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:23.165256 28910 hybrid_clock.cc:648] HybridClock initialized: now 1786223423165235 us; error 36 us; skew 500 ppm
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:23.167699 28910 init.cc:377] Logged in from keytab as kudu/127.4.177.125@KRBTEST.COM (short username kudu)
I20260808 21:10:23.168068 28910 webserver.cc:460] Webserver started at http://127.4.177.125:39141/ using document root <none> and password file <none>
I20260808 21:10:23.168291 28910 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:23.168339 28910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:23.168462 28910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:23.169515 28910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data/instance:
uuid: "f2a784d080104bfa9ea37efad873da24"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "6525036f185b644ccd41e7791478c095"
server_key_iv: "ed58c34bad3292a4ccc94503ffa88828"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.169871 28910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal/instance:
uuid: "f2a784d080104bfa9ea37efad873da24"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "6525036f185b644ccd41e7791478c095"
server_key_iv: "ed58c34bad3292a4ccc94503ffa88828"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.171227 28910 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260808 21:10:23.171945 28925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:23.172112 28910 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:23.172219 28910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "f2a784d080104bfa9ea37efad873da24"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "6525036f185b644ccd41e7791478c095"
server_key_iv: "ed58c34bad3292a4ccc94503ffa88828"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.172288 28910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:23.178352 28902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } attempt: 2
I20260808 21:10:23.178785 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.178653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56080)
0808 21:10:23.178711 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.178749 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 21:10:23.179389 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.179099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56086)
0808 21:10:23.179297 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.179340 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260808 21:10:23.179479 28902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:34521: Network error: Client connection negotiation failed: client connection to 127.4.177.125:34521: connect: Connection refused (error 111)
I20260808 21:10:23.180888 28910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:23.182577 28910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:23.182683 28910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:23.186583 28910 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.125:34521
I20260808 21:10:23.186619 28981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.125:34521 every 8 connection(s)
I20260808 21:10:23.186954 28910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
I20260808 21:10:23.187968 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:23.189095 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28910
I20260808 21:10:23.189179 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-1/wal/instance
I20260808 21:10:23.189375 4805 external_mini_cluster.cc:1428] Setting key 4f0f294532714e66e76bcd533e52eabf
I20260808 21:10:23.189533 28982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.124@KRBTEST.COM; defaulting to no policy
I20260808 21:10:23.198096 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.190208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50134 (local address 127.4.177.126:44311)
0808 21:10:23.190277 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.190281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.190493 (+ 212us) server_negotiation.cc:407] Connection header received
0808 21:10:23.190975 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.190976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.190987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.191045 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.191689 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.192156 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.193073 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.193272 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.193343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.195381 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.195383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.195383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.195390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.196753 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.197152 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.197154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.197154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.197212 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.197466 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.197467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.197468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.197579 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.197653 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.197924 (+ 271us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.198004 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:10:23.198288 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.190206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44311 (local address 127.0.0.1:50134)
0808 21:10:23.190379 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.190413 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.190876 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.191116 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.191119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.191424 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.191599 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.191603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.192248 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.192248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.192976 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.192977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.193391 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.193392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.193489 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.193955 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.193969 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.195282 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.196837 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.196844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.196847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.197062 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.197283 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.197284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.197284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.197396 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.197659 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.197666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.197798 (+ 132us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.197903 (+ 105us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.198049 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260808 21:10:23.198789 28982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.124@KRBTEST.COM" created.
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:23.206161 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.199063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56098)
0808 21:10:23.199108 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.199117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.199201 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.199407 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.199408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.199562 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.199603 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.199604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.200700 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.200701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.201174 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.201176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.201646 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.201647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.201707 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.202110 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.202114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.203417 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.205134 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.205135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.205135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.205291 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.205499 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.205500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.205500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.205550 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.205853 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.205854 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.205892 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.205958 (+ 66us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.206055 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260808 21:10:23.206249 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.199098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.4.177.125:34521)
0808 21:10:23.199263 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.199268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.199286 (+ 18us) server_negotiation.cc:407] Connection header received
0808 21:10:23.199309 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.199311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.199325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.199393 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.199679 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.200617 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.201267 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.201576 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.201649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.203511 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.203519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.203520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.203526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.205040 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.205379 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.205381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.205382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.205435 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.205640 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.205641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.205641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.205773 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.205857 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.205999 (+ 142us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.206104 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260808 21:10:23.206732 28982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:23.207095 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.206970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48254)
0808 21:10:23.207009 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.207062 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260808 21:10:23.207271 28930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
I20260808 21:10:23.207660 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.207595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48260)
0808 21:10:23.207619 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.207642 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260808 21:10:23.207782 28982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39219: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
I20260808 21:10:23.216001 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:39219
--webserver_interface=127.4.177.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:10:23.257004 28982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } attempt: 1
I20260808 21:10:23.257521 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.257379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48274)
0808 21:10:23.257445 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.257490 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 21:10:23.257864 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.257767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48288)
0808 21:10:23.257811 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.257836 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 21:10:23.257961 28982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39219: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
I20260808 21:10:23.270661 28902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } attempt: 3
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:23.278025 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.271102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.4.177.125:34521)
0808 21:10:23.271183 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.271186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.271199 (+ 13us) server_negotiation.cc:407] Connection header received
0808 21:10:23.271296 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.271297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.271311 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.271383 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.271726 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.272239 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.272958 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.273198 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.273274 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.275453 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.275454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.275454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.275461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.276883 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.277307 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.277309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.277309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.277370 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.277580 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.277581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.277581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.277687 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.277738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.277893 (+ 155us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.277959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 21:10:23.278076 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.271020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56108)
0808 21:10:23.271083 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.271097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.271208 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.271389 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.271390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.271565 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.271643 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.271644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.272335 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.272337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.272862 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.272863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.273285 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.273286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.273363 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.273858 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.273863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.275367 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.276969 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.276970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.276971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.277206 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.277447 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.277448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.277448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.277517 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.277749 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.277749 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.277800 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.277877 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.277976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 21:10:23.278482 28902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:23.278904 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.278785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48298)
0808 21:10:23.278838 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.278877 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:10:23.279264 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.279163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48312)
0808 21:10:23.279212 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.279244 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 21:10:23.279376 28902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39219: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
W20260808 21:10:23.301038 28990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:23.301258 28990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:23.301276 28990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:23.301323 28990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:23.302479 28990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:23.302521 28990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:23.302536 28990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:23.302547 28990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:23.302559 28990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:23.304169 28990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:44311,127.4.177.125:34521,127.4.177.124:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.177.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:23.304418 28990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:23.304661 28990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:23.309540 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:23.309571 28995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:23.309540 28996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:23.309793 28990 server_base.cc:1034] running on GCE node
I20260808 21:10:23.310071 28990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:23.310372 28990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:23.310518 28902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } attempt: 1
I20260808 21:10:23.310918 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.310793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48318)
0808 21:10:23.310855 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.310889 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260808 21:10:23.311292 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.311219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48322)
0808 21:10:23.311251 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.311274 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260808 21:10:23.311395 28902 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39219: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
I20260808 21:10:23.311518 28990 hybrid_clock.cc:648] HybridClock initialized: now 1786223423311490 us; error 43 us; skew 500 ppm
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:23.314050 28990 init.cc:377] Logged in from keytab as kudu/127.4.177.124@KRBTEST.COM (short username kudu)
I20260808 21:10:23.314492 28990 webserver.cc:460] Webserver started at http://127.4.177.124:44769/ using document root <none> and password file <none>
I20260808 21:10:23.314699 28990 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:23.314741 28990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:23.314826 28990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:23.315860 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data/instance:
uuid: "b44afc71266447d69772fc1c128aee1a"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "04ba037d78530fc392c82da68b3bf4fc"
server_key_iv: "c58a837960dddcf0db056e6ef629b479"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.316155 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal/instance:
uuid: "b44afc71266447d69772fc1c128aee1a"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "04ba037d78530fc392c82da68b3bf4fc"
server_key_iv: "c58a837960dddcf0db056e6ef629b479"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.317420 28990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:23.318114 29006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:23.318292 28990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:23.318398 28990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "b44afc71266447d69772fc1c128aee1a"
format_stamp: "Formatted at 2026-08-08 21:10:23 on dist-test-slave-chsk"
server_key: "04ba037d78530fc392c82da68b3bf4fc"
server_key_iv: "c58a837960dddcf0db056e6ef629b479"
server_key_version: "encryptionkey@0"
I20260808 21:10:23.318473 28990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:23.326112 28982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } attempt: 2
I20260808 21:10:23.326550 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.326406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48338)
0808 21:10:23.326470 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.326514 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:10:23.326951 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.326855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48348)
0808 21:10:23.326902 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.326930 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 21:10:23.327097 28982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:39219: Network error: Client connection negotiation failed: client connection to 127.4.177.124:39219: connect: Connection refused (error 111)
I20260808 21:10:23.331728 28990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:23.334194 28990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:23.334362 28990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:23.339274 28990 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.124:39219
I20260808 21:10:23.339293 29060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.124:39219 every 8 connection(s)
I20260808 21:10:23.339686 28990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
I20260808 21:10:23.340798 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 28990
I20260808 21:10:23.340919 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786223155551886-4805-0/minicluster-data/master-2/wal/instance
I20260808 21:10:23.340965 29061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:23.341149 4805 external_mini_cluster.cc:1428] Setting key 2e902957527925e9b8e2078ca111ded6
I20260808 21:10:23.343447 29061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:10:23.356372 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.344269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.4.177.126:44311)
0808 21:10:23.345468 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.345471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.345482 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:10:23.345516 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.345517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.345526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.345582 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.346474 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.347260 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.348658 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.348957 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.349028 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.352811 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.352812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.352812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.352820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.354648 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.355257 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.355258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.355258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.355316 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.355639 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.355640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.355640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.355810 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.355887 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.356164 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.356277 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167}
I20260808 21:10:23.356532 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.344192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:44311 (local address 127.0.0.1:50150)
0808 21:10:23.344327 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.344367 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.344908 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.345715 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.345721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.346121 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.346368 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.346374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.347358 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.347359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.348561 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.348564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.349070 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.349071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.349205 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.350134 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.350155 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.352705 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.354771 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.354778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.354781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.355163 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.355413 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.355414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.355414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.355558 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.355893 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.355899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.356016 (+ 117us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.356144 (+ 128us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.356316 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260808 21:10:23.357263 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.343396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.4.177.124:39219)
0808 21:10:23.345662 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.345666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.345682 (+ 16us) server_negotiation.cc:407] Connection header received
0808 21:10:23.345712 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.345713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.345737 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.345836 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.346538 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.347867 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.348691 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.349155 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.349246 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.353317 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.353326 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.353327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.353333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.355449 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.355932 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.355935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.355936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.355993 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.356243 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.356244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.356245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.356416 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.356552 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.357048 (+ 496us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.357137 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2228,"thread_start_us":80,"threads_started":1}
I20260808 21:10:23.357710 29061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:23.366533 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.357993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:34521 (local address 127.0.0.1:56112)
0808 21:10:23.358037 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.358050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.358092 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.358292 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.358293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.358453 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.358497 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.358498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.359352 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.359353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.360071 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.360073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.360570 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.360571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.360634 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.361094 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.361099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.362710 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.365345 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.365346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.365347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.365539 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.365787 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.365788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.365788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.365845 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.366130 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.366131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.366192 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.366303 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.366407 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260808 21:10:23.366546 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.358107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.4.177.125:34521)
0808 21:10:23.358165 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.358167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.358176 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:10:23.358202 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.358203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.358215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.358273 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.358589 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.359276 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.360154 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.360485 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.360564 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.362805 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.362807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.362807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.362815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.365246 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.365646 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.365648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.365648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.365708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.365904 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.365905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.365906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.366061 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.366138 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.366340 (+ 202us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.366444 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:10:23.367252 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.358113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.4.177.126:44311)
0808 21:10:23.358149 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.358150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.358226 (+ 76us) server_negotiation.cc:407] Connection header received
0808 21:10:23.358317 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.358318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.358327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.358357 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.358972 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.359888 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.360901 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.361208 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.361284 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.364128 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.364130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.364130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.364137 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.365838 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.366188 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.366190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.366190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.366269 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.366492 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.366493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.366493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.366630 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.366812 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.367041 (+ 229us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.367117 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:10:23.367321 29061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:23.374893 29065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.367607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48368)
0808 21:10:23.367766 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.367776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.367815 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.367963 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.367964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.368127 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.368160 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.368162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.368985 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.368986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.369457 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.369458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.369950 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.369951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.369994 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.370420 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.370424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.371871 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.373811 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.373812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.373812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.373971 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.374204 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.374204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.374205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.374279 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.374550 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.374551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.374592 (+ 41us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.374670 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.374775 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260808 21:10:23.374903 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.367642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.4.177.124:39219)
0808 21:10:23.367695 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.367697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.367861 (+ 164us) server_negotiation.cc:407] Connection header received
0808 21:10:23.367878 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.367878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.367890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.367930 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.368232 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.368909 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.369561 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.369851 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.369909 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.371991 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.371992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.371993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.372001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.373727 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.374041 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.374042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.374042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.374096 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.374327 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.374328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.374328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.374484 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.374548 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.374708 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.374822 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":32}
I20260808 21:10:23.377012 29061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: Bootstrap starting.
I20260808 21:10:23.377091 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.368024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.4.177.125:34521)
0808 21:10:23.368061 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.368063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.368073 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:10:23.368097 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.368098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.368109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.368170 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.368717 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.369430 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.370343 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.370659 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.370703 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.373553 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.373555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.373555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.373563 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.375850 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.376208 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.376210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.376210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.376282 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.376500 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.376501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.376501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.376653 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.376839 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.376883 (+ 44us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.376965 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260808 21:10:23.378322 29061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:23.378691 29061 log.cc:824] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:23.379777 29061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: No bootstrap required, opened a new log
I20260808 21:10:23.380589 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:23.381733 29061 raft_consensus.cc:348] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.381880 29061 raft_consensus.cc:729] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44afc71266447d69772fc1c128aee1a, State: Initialized, Role: FOLLOWER
I20260808 21:10:23.381968 29061 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.382340 29061 sys_catalog.cc:564] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:23.382308 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.382408 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.381809 28875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "61085ebf460442fea76ab678d023dc0e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50154
W20260808 21:10:23.384176 29078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:23.384230 29078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:23.384451 28990 master_runner.cc:186] Error getting master registration for 127.4.177.126:44311: OK, instance_id { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" instance_seqno: 1786223423042509 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:10:23.387638 28990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:10:23.387710 28902 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } attempt: 2
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223422, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:10:23.400921 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.387956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48372)
0808 21:10:23.388007 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.388018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.388064 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.388225 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.388226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.388391 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.388432 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.388434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.389311 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.389312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.389852 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.389854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.390356 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.390357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.390426 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.395528 (+ 5102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.395534 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.397045 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.399612 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.399613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.399613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.399839 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.400113 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.400114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.400114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.400187 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.400602 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.400603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.400669 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.400715 (+ 46us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.400818 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:10:23.401430 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.388041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.4.177.124:39219)
0808 21:10:23.388094 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.388096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.388104 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:10:23.388141 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.388141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.388157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.388207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.388522 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.389240 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.389942 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.390287 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.390350 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.397152 (+ 6802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.397154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.397154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.397194 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.399506 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.399928 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.399929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.399929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.400027 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.400265 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.400266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.400266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.400511 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.401133 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.401200 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.401331 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260808 21:10:23.403174 28902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e: Bootstrap starting.
I20260808 21:10:23.403981 28902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:23.404310 28902 log.cc:824] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:23.405165 28902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e: No bootstrap required, opened a new log
I20260808 21:10:23.406176 28902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.406313 28902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61085ebf460442fea76ab678d023dc0e, State: Initialized, Role: FOLLOWER
I20260808 21:10:23.406401 28902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.406682 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.406751 28902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:23.406821 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.406827 28875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 21:10:23.407002 28875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:23.407868 28875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.408115 28875 leader_election.cc:288] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [CANDIDATE]: Term 1 election: Requested vote from peers f2a784d080104bfa9ea37efad873da24 (127.4.177.125:34521), b44afc71266447d69772fc1c128aee1a (127.4.177.124:39219)
I20260808 21:10:23.408438 29029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61085ebf460442fea76ab678d023dc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b44afc71266447d69772fc1c128aee1a"
I20260808 21:10:23.408584 29029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:23.409583 29029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61085ebf460442fea76ab678d023dc0e in term 1.
I20260808 21:10:23.409901 28850 leader_election.cc:302] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61085ebf460442fea76ab678d023dc0e, b44afc71266447d69772fc1c128aee1a; no voters:
I20260808 21:10:23.410022 29081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:23.410219 29081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [term 1 LEADER]: Becoming Leader. State: Replica: 61085ebf460442fea76ab678d023dc0e, State: Running, Role: LEADER
I20260808 21:10:23.410310 29081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.410727 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61085ebf460442fea76ab678d023dc0e. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.410795 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:23.409680 28832 master_runner.cc:186] Error getting master registration for 127.4.177.125:34521: OK, instance_id { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" instance_seqno: 1786223423185472 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:10:23.412662 28832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:10:23.413192 29101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:23.413687 29101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:23.418268 28982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } attempt: 3
I20260808 21:10:23.418574 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.409482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.4.177.126:44311)
0808 21:10:23.409522 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.409524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.410465 (+ 941us) server_negotiation.cc:407] Connection header received
0808 21:10:23.410621 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.410622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.410644 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.410675 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.411624 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.412346 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.413399 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.413672 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.413743 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.415549 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.415551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.415551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.415558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.417357 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.417746 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.417747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.417747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.417799 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.418001 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.418002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.418002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.418125 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.418197 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.418414 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.418488 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Aug 08 21:10:23 dist-test-slave-chsk krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223423, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:10:23.425755 29029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [term 1 FOLLOWER]: Refusing update from remote peer 61085ebf460442fea76ab678d023dc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:10:23.426060 29098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:10:23.427595 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61085ebf460442fea76ab678d023dc0e. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.427685 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.429617 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.418503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:39219 (local address 127.0.0.1:48392)
0808 21:10:23.418575 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:23.418584 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:23.418617 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:23.419537 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:23.419538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:23.419731 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:23.419774 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.419776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.420719 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.420720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.421492 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:23.421494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.423256 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.423258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:23.423313 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.423742 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:23.423747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:23.425191 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:23.428473 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.428474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.428474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.428698 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.428927 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:23.428927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:23.428928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:23.428988 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:23.429290 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:23.429291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:23.429335 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 21:10:23.429394 (+ 59us) client_negotiation.cc:240] Negotiation successful
0808 21:10:23.429495 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 21:10:23.430025 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.418679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.4.177.124:39219)
0808 21:10:23.419399 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.419401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.419407 (+ 6us) server_negotiation.cc:407] Connection header received
0808 21:10:23.419426 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.419426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.419436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.419840 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.419865 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.420621 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.422839 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.423175 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.425388 (+ 2213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.425438 (+ 50us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.425439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.425439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.425444 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.428383 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.428787 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.428788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.428789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.428849 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.429059 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.429060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.429060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.429210 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.429791 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.429840 (+ 49us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.429927 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":93,"threads_started":1}
I20260808 21:10:23.430781 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.410077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.4.177.125:34521)
0808 21:10:23.410120 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.410122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.415844 (+ 5722us) server_negotiation.cc:407] Connection header received
0808 21:10:23.416003 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.416004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.416015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.416060 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.417167 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.417872 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.420448 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.420772 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.420808 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.426482 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.426483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.426484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.426491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.428371 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.429182 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.429183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.429183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.429234 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.429552 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.429553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.429553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.429687 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.429776 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.430637 (+ 861us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.430710 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260808 21:10:23.431667 28982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: Bootstrap starting.
I20260808 21:10:23.432451 28982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:23.432806 28982 log.cc:824] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:23.434446 29098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.434549 29098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:23.434746 29104 mvcc.cc:205] Tried to move back new op lower bound from 7316371142347083776 to 7316371142289387520. Current Snapshot: MvccSnapshot[applied={T|T < 7316371142347083776}]
I20260808 21:10:23.434876 28982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: No bootstrap required, opened a new log
I20260808 21:10:23.435869 29101 catalog_manager.cc:1347] Generated new cluster ID: cf32fcf4085943c699fe868aa3f4d34c
I20260808 21:10:23.435917 29101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:23.437258 28982 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.437402 28982 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a784d080104bfa9ea37efad873da24, State: Initialized, Role: FOLLOWER
I20260808 21:10:23.437484 28982 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } }
I20260808 21:10:23.437824 28982 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:23.437885 28954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "61085ebf460442fea76ab678d023dc0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f2a784d080104bfa9ea37efad873da24"
I20260808 21:10:23.437974 28954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:23.438922 28954 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61085ebf460442fea76ab678d023dc0e in term 1.
I20260808 21:10:23.439170 28953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [term 1 FOLLOWER]: Refusing update from remote peer 61085ebf460442fea76ab678d023dc0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:10:23.439512 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.439599 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.439605 29081 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:10:23.439888 29106 mvcc.cc:205] Tried to move back new op lower bound from 7316371142347083776 to 7316371142289387520. Current Snapshot: MvccSnapshot[applied={T|T < 7316371142347083776}]
W20260808 21:10:23.440577 29121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:23.440665 29121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:23.442377 29129 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61085ebf460442fea76ab678d023dc0e. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.442502 29129 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.442598 29129 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.442680 29129 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.443244 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.443367 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:23.443754 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:23.415425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.4.177.124:39219)
0808 21:10:23.415463 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.415465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.419370 (+ 3905us) server_negotiation.cc:407] Connection header received
0808 21:10:23.419649 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.419650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.419674 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.419703 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.422202 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.422907 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.426360 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:23.426652 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:23.426700 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:23.433957 (+ 7257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:23.433958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:23.433958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:23.433965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:23.436046 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.436875 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.436876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.436877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.436938 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:23.437299 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:23.437301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:23.437301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:23.437433 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:23.437536 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:23.443588 (+ 6052us) server_negotiation.cc:299] Negotiation successful
0808 21:10:23.443676 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260808 21:10:23.461186 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61085ebf460442fea76ab678d023dc0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61085ebf460442fea76ab678d023dc0e" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 44311 } } peers { permanent_uuid: "f2a784d080104bfa9ea37efad873da24" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 34521 } } peers { permanent_uuid: "b44afc71266447d69772fc1c128aee1a" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 39219 } } }
I20260808 21:10:23.461323 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:23.463814 29101 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:23.464558 29101 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:23.474078 29101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61085ebf460442fea76ab678d023dc0e: Generated new TSK 0
I20260808 21:10:23.474303 29101 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 21:10:23.496016 29064 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.494308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.4.177.124:39219)
0808 21:10:23.494355 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.494357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.495348 (+ 991us) server_negotiation.cc:407] Connection header received
0808 21:10:23.495454 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.495456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.495468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.495499 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.495946 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48404: BlockingRecv error: recv got EOF from 127.0.0.1:48404 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 21:10:23.496799 28904 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.494732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.4.177.126:44311)
0808 21:10:23.494773 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.494775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.496276 (+ 1501us) server_negotiation.cc:407] Connection header received
0808 21:10:23.496372 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.496373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.496393 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.496423 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.496737 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50184: BlockingRecv error: recv got EOF from 127.0.0.1:50184 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260808 21:10:23.497525 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.495046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.4.177.125:34521)
0808 21:10:23.495087 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.495088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.496993 (+ 1905us) server_negotiation.cc:407] Connection header received
0808 21:10:23.497129 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.497130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.497141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.497171 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.497464 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56148: BlockingRecv error: recv got EOF from 127.0.0.1:56148 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 21:10:23.499886 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.498780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.4.177.125:34521)
0808 21:10:23.498814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.498816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.499366 (+ 550us) server_negotiation.cc:407] Connection header received
0808 21:10:23.499490 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.499491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.499509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.499537 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.499842 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56152: BlockingRecv error: recv got EOF from 127.0.0.1:56152 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 21:10:23.502861 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.502162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.4.177.125:34521)
0808 21:10:23.502194 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.502196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.502399 (+ 203us) server_negotiation.cc:407] Connection header received
0808 21:10:23.502489 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.502490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.502500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.502528 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.502819 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56164: BlockingRecv error: recv got EOF from 127.0.0.1:56164 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 21:10:23.506686 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.505973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.4.177.125:34521)
0808 21:10:23.506004 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.506006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.506223 (+ 217us) server_negotiation.cc:407] Connection header received
0808 21:10:23.506312 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.506313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.506322 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.506352 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.506647 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260808 21:10:23.512712 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.511905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.4.177.125:34521)
0808 21:10:23.511938 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.511940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.512221 (+ 281us) server_negotiation.cc:407] Connection header received
0808 21:10:23.512316 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.512317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.512327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.512355 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.512667 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56184: BlockingRecv error: recv got EOF from 127.0.0.1:56184 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 21:10:23.522974 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.522053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.4.177.125:34521)
0808 21:10:23.522120 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.522122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.522379 (+ 257us) server_negotiation.cc:407] Connection header received
0808 21:10:23.522488 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.522489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.522501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.522532 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.522920 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56200: BlockingRecv error: recv got EOF from 127.0.0.1:56200 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 21:10:23.541528 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.540491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.4.177.125:34521)
0808 21:10:23.540537 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.540539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.540877 (+ 338us) server_negotiation.cc:407] Connection header received
0808 21:10:23.540992 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.540994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.541006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.541050 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.541468 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56202: BlockingRecv error: recv got EOF from 127.0.0.1:56202 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 21:10:23.576215 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.575185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.4.177.125:34521)
0808 21:10:23.575235 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.575237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.575501 (+ 264us) server_negotiation.cc:407] Connection header received
0808 21:10:23.575604 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.575605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.575617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.575648 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.576151 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56204: BlockingRecv error: recv got EOF from 127.0.0.1:56204 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260808 21:10:23.642448 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.641657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.4.177.125:34521)
0808 21:10:23.641732 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.641734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.641874 (+ 140us) server_negotiation.cc:407] Connection header received
0808 21:10:23.641922 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.641923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.641933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.641985 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.642359 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56220: BlockingRecv error: recv got EOF from 127.0.0.1:56220 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260808 21:10:23.772454 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:23.771778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.4.177.125:34521)
0808 21:10:23.771844 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:23.771851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:23.771863 (+ 12us) server_negotiation.cc:407] Connection header received
0808 21:10:23.771935 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:23.771935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:23.771946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:23.772011 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:23.772362 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56224: BlockingRecv error: recv got EOF from 127.0.0.1:56224 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260808 21:10:24.030465 28987 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:24.029792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56228 (local address 127.4.177.125:34521)
0808 21:10:24.029870 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.029872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.029883 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:10:24.029937 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.029938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.029948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.030001 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:24.030390 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56228: BlockingRecv error: recv got EOF from 127.0.0.1:56228 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260808 21:10:24.384698 29078 catalog_manager.cc:1259] Loaded cluster ID: cf32fcf4085943c699fe868aa3f4d34c
I20260808 21:10:24.384784 29078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: loading cluster ID for follower catalog manager: success
I20260808 21:10:24.385568 29078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: acquiring CA information for follower catalog manager: success
I20260808 21:10:24.385836 29078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b44afc71266447d69772fc1c128aee1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:10:24.441130 29121 catalog_manager.cc:1259] Loaded cluster ID: cf32fcf4085943c699fe868aa3f4d34c
I20260808 21:10:24.441227 29121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: loading cluster ID for follower catalog manager: success
I20260808 21:10:24.442111 29121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: acquiring CA information for follower catalog manager: success
I20260808 21:10:24.442456 29121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f2a784d080104bfa9ea37efad873da24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:10:24.546072 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:24.543677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.4.177.125:34521)
0808 21:10:24.543813 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.543815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.543879 (+ 64us) server_negotiation.cc:407] Connection header received
0808 21:10:24.543923 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.543923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.543975 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.544054 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:24.544198 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.544694 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.545396 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.545579 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.545631 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:24.545784 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:24.545889 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:24.545946 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:24.545986 (+ 40us) server_negotiation.cc:299] Negotiation successful
0808 21:10:24.546003 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260808 21:10:24.550163 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:24.547960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.4.177.124:39219)
0808 21:10:24.548073 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.548075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.548086 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:10:24.548171 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.548171 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.548203 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.548271 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:24.548381 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.548870 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.549562 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.549746 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.549794 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:24.549914 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:24.550000 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:24.550051 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:24.550087 (+ 36us) server_negotiation.cc:299] Negotiation successful
0808 21:10:24.550103 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260808 21:10:24.554601 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:24.551845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.4.177.124:39219)
0808 21:10:24.551912 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.551914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.552144 (+ 230us) server_negotiation.cc:407] Connection header received
0808 21:10:24.552212 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.552213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.552223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.552275 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:24.552436 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.552910 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.553806 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.554096 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.554170 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:24.554331 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:24.554360 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:24.554433 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:24.554494 (+ 61us) server_negotiation.cc:299] Negotiation successful
0808 21:10:24.554513 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 21:10:24.555075 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:24.551951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.4.177.125:34521)
0808 21:10:24.552002 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.552003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.552312 (+ 309us) server_negotiation.cc:407] Connection header received
0808 21:10:24.552344 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.552345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.552356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.552404 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:24.552614 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.553319 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.554280 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.554566 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.554624 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:24.554801 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:24.554829 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:24.554857 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:24.554981 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 21:10:24.555008 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260808 21:10:24.555184 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:24.551808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.4.177.126:44311)
0808 21:10:24.551920 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:24.551922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:24.552113 (+ 191us) server_negotiation.cc:407] Connection header received
0808 21:10:24.552204 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:24.552205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:24.552259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:24.552352 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:24.552509 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.553302 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.554167 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:24.554478 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:24.554539 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:24.554795 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:24.554907 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:24.554963 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:24.555046 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 21:10:24.555075 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260808 21:10:24.556736 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28832
I20260808 21:10:24.564118 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28910
I20260808 21:10:24.569861 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 28990
2026-08-08T21:10:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2843 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 21:10:24 dist-test-slave-chsk krb5kdc[29218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 21:10:24 dist-test-slave-chsk krb5kdc[29218](info): set up 2 sockets
Aug 08 21:10:24 dist-test-slave-chsk krb5kdc[29218](info): commencing operation
krb5kdc: starting...
W20260808 21:10:25.638933 4805 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/test-user.keytab.
Aug 08 21:10:25 dist-test-slave-chsk krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T21:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T21:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.177.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
Entry for principal kudu/127.4.177.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab.
I20260808 21:10:25.761904 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38123
--webserver_interface=127.4.177.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 21:10:25.842710 29231 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:25.842962 29231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:25.842993 29231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:25.843034 29231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:25.844219 29231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:25.844286 29231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:25.844308 29231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:25.844328 29231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:25.844359 29231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:25.845783 29231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.126.keytab
--principal=kudu/127.4.177.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.126:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.177.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:25.846071 29231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:25.846315 29231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:25.851061 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:25.851130 29236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:25.851089 29237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:25.851311 29231 server_base.cc:1034] running on GCE node
I20260808 21:10:25.851634 29231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:25.851948 29231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:25.853101 29231 hybrid_clock.cc:648] HybridClock initialized: now 1786223425853070 us; error 51 us; skew 500 ppm
Aug 08 21:10:25 dist-test-slave-chsk krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:25.855646 29231 init.cc:377] Logged in from keytab as kudu/127.4.177.126@KRBTEST.COM (short username kudu)
I20260808 21:10:25.856098 29231 webserver.cc:460] Webserver started at http://127.4.177.126:40481/ using document root <none> and password file <none>
I20260808 21:10:25.856360 29231 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:25.856429 29231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:25.856577 29231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:25.857615 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data/instance:
uuid: "d6ffb76082a14ad5b793e05a69fe0fca"
format_stamp: "Formatted at 2026-08-08 21:10:25 on dist-test-slave-chsk"
server_key: "104cebf50798340237573e5825856556"
server_key_iv: "9aa900793c1fef1fd3caaf3f041dc5ff"
server_key_version: "encryptionkey@0"
I20260808 21:10:25.857985 29231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance:
uuid: "d6ffb76082a14ad5b793e05a69fe0fca"
format_stamp: "Formatted at 2026-08-08 21:10:25 on dist-test-slave-chsk"
server_key: "104cebf50798340237573e5825856556"
server_key_iv: "9aa900793c1fef1fd3caaf3f041dc5ff"
server_key_version: "encryptionkey@0"
I20260808 21:10:25.859365 29231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260808 21:10:25.860106 29246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:25.860293 29231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:25.860399 29231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal
uuid: "d6ffb76082a14ad5b793e05a69fe0fca"
format_stamp: "Formatted at 2026-08-08 21:10:25 on dist-test-slave-chsk"
server_key: "104cebf50798340237573e5825856556"
server_key_iv: "9aa900793c1fef1fd3caaf3f041dc5ff"
server_key_version: "encryptionkey@0"
I20260808 21:10:25.860471 29231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:25.880568 29231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:25.882447 29231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:25.882618 29231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:25.886917 29231 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.126:38123
I20260808 21:10:25.886976 29300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.126:38123 every 8 connection(s)
I20260808 21:10:25.887388 29231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/data/info.pb
I20260808 21:10:25.888338 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:25.889941 29301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:25 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:25.896997 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 29231
I20260808 21:10:25.897149 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-0/wal/instance
I20260808 21:10:25.897385 4805 external_mini_cluster.cc:1428] Setting key 3a66c1df2db21e281d7d14720faf4f7c
I20260808 21:10:25.899444 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.890526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38123 (local address 127.0.0.1:33080)
0808 21:10:25.890703 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:25.890744 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:25.891293 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:25.891554 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:25.891557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:25.891800 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:25.891952 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:25.891956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:25.892965 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:25.892966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:25.893625 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:25.893627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:25.894091 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:25.894092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:25.894142 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:25.894574 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:25.894581 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:25.895987 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:25.897907 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:25.897914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:25.897916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:25.898100 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:25.898355 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:25.898356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:25.898357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:25.898441 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:25.898795 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:25.898800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:25.898860 (+ 60us) client_negotiation.cc:769] Sending connection context
0808 21:10:25.898973 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 21:10:25.899264 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":147,"thread_start_us":88,"threads_started":1}
I20260808 21:10:25.899449 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.890580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.4.177.126:38123)
0808 21:10:25.890771 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:25.890775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:25.890830 (+ 55us) server_negotiation.cc:407] Connection header received
0808 21:10:25.891428 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:25.891430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:25.891446 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:25.891525 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:25.892033 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:25.892865 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:25.893707 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:25.893999 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:25.894073 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:25.896063 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:25.896070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:25.896070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:25.896076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:25.897812 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:25.898200 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:25.898203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:25.898203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:25.898286 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:25.898507 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:25.898508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:25.898508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:25.898724 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:25.898817 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:25.899021 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 21:10:25.899177 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260808 21:10:25.900151 29301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:25.900483 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.900384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47882)
0808 21:10:25.900422 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:25.900463 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260808 21:10:25.900651 29248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
I20260808 21:10:25.901078 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.900991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47890)
0808 21:10:25.901038 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:25.901062 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260808 21:10:25.901196 29301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:32873: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.177.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
Entry for principal kudu/127.4.177.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab.
I20260808 21:10:25.924331 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:32873
--webserver_interface=127.4.177.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:10:25.950428 29301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } attempt: 1
I20260808 21:10:25.950974 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.950829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47900)
0808 21:10:25.950903 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:25.950945 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 21:10:25.951375 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:25.951290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47906)
0808 21:10:25.951333 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:25.951358 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 21:10:25.951483 29301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:32873: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
W20260808 21:10:26.008188 29308 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:26.008399 29308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:26.008416 29308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:26.008435 29308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:26.009653 29308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:26.009692 29308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:26.009706 29308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:26.009719 29308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:26.009766 29308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:26.011314 29308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.125.keytab
--principal=kudu/127.4.177.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.125:32873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.177.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:26.011562 29308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:26.011792 29308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:26.016592 29314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:26.016747 29308 server_base.cc:1034] running on GCE node
W20260808 21:10:26.016623 29315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:26.016590 29317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:26.017077 29308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:26.017367 29308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:26.018517 29308 hybrid_clock.cc:648] HybridClock initialized: now 1786223426018495 us; error 37 us; skew 500 ppm
I20260808 21:10:26.019618 29301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } attempt: 2
I20260808 21:10:26.019989 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.019869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47916)
0808 21:10:26.019927 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.019962 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260808 21:10:26.020324 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.020228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47930)
0808 21:10:26.020271 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.020295 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:10:26.020411 29301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.125:32873: Network error: Client connection negotiation failed: client connection to 127.4.177.125:32873: connect: Connection refused (error 111)
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:26.021337 29308 init.cc:377] Logged in from keytab as kudu/127.4.177.125@KRBTEST.COM (short username kudu)
I20260808 21:10:26.021793 29308 webserver.cc:460] Webserver started at http://127.4.177.125:35785/ using document root <none> and password file <none>
I20260808 21:10:26.021975 29308 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:26.022028 29308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:26.022115 29308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:26.023128 29308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data/instance:
uuid: "d9f94941782a4972b67c04c3eaf03750"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "ee475d8ec2e82ffa0a7a62c1e64c82cf"
server_key_iv: "3c397abfb1d6a475ea2be436459462c9"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.023504 29308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal/instance:
uuid: "d9f94941782a4972b67c04c3eaf03750"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "ee475d8ec2e82ffa0a7a62c1e64c82cf"
server_key_iv: "3c397abfb1d6a475ea2be436459462c9"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.024746 29308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260808 21:10:26.025431 29325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:26.025585 29308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260808 21:10:26.025676 29308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal
uuid: "d9f94941782a4972b67c04c3eaf03750"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "ee475d8ec2e82ffa0a7a62c1e64c82cf"
server_key_iv: "3c397abfb1d6a475ea2be436459462c9"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.025730 29308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:26.061273 29308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:26.063272 29308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:26.063422 29308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:26.067538 29308 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.125:32873
I20260808 21:10:26.067600 29380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.125:32873 every 8 connection(s)
I20260808 21:10:26.067952 29308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/data/info.pb
I20260808 21:10:26.068931 29381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:26.070354 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 29308
I20260808 21:10:26.070451 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-1/wal/instance
I20260808 21:10:26.070654 4805 external_mini_cluster.cc:1428] Setting key c46d77a4e8c205d0205048ebcc66a8e5
I20260808 21:10:26.070546 29381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.177.124@KRBTEST.COM; defaulting to no policy
I20260808 21:10:26.080324 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.071379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.4.177.126:38123)
0808 21:10:26.071446 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.071448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.071675 (+ 227us) server_negotiation.cc:407] Connection header received
0808 21:10:26.072118 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.072120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.072135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.072218 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.073050 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.073581 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.074560 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.074922 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.075004 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.077201 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.077203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.077203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.077212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.078963 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.079477 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.079478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.079479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.079541 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.079761 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.079761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.079762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.079861 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.079907 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.080141 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.080257 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260808 21:10:26.080579 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.071362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38123 (local address 127.0.0.1:33088)
0808 21:10:26.071536 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.071577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.072033 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.072265 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.072268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.072663 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.072952 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.072958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.073682 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.073683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.074470 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.074472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.075018 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.075019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.075124 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.075712 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.075727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.077093 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.079058 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.079065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.079067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.079404 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.079620 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.079621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.079621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.079719 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.079911 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.079916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.080031 (+ 115us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.080142 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.080345 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260808 21:10:26.081192 29381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.177.124@KRBTEST.COM" created.
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:26.088918 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.081463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47940)
0808 21:10:26.081515 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.081528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.081570 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.081877 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.081877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.082027 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.082073 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.082074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.082931 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.082932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.083614 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.083617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.084089 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.084091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.084177 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.084550 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.084556 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.085706 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.087894 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.087895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.087896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.088059 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.088291 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.088291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.088292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.088342 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.088686 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.088687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.088721 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.088771 (+ 50us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.088851 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260808 21:10:26.089044 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.081569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.4.177.125:32873)
0808 21:10:26.081742 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.081746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.081761 (+ 15us) server_negotiation.cc:407] Connection header received
0808 21:10:26.081778 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.081779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.081789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.081864 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.082158 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.082828 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.083708 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.084011 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.084072 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.085792 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.085800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.085801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.085808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.087809 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.088143 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.088147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.088147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.088217 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.088407 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.088408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.088408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.088607 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.088708 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.088809 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.088927 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260808 21:10:26.089643 29381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:26.090013 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.089877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38884)
0808 21:10:26.089934 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.089984 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260808 21:10:26.090152 29327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
I20260808 21:10:26.090631 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.090504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38900)
0808 21:10:26.090558 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.090590 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260808 21:10:26.090752 29381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:36409: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
Entry for principal kudu/127.4.177.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab.
I20260808 21:10:26.098014 4805 external_mini_cluster.cc:1326] Running /tmp/dist-test-task496WfL/build/release/bin/kudu
/tmp/dist-test-task496WfL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:36409
--webserver_interface=127.4.177.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.177.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 21:10:26.111544 29301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } attempt: 3
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:26.118382 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.111945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.4.177.125:32873)
0808 21:10:26.112008 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.112010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.112021 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:10:26.112062 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.112068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.112079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.112131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.112463 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.112957 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.113666 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.113888 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.113976 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.115877 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.115878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.115879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.115886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.117244 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.117599 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.117601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.117601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.117663 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.117862 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.117863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.117863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.117977 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.118053 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.118206 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.118285 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 21:10:26.118383 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.111868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47954)
0808 21:10:26.111925 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.111938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.111993 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.112159 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.112160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.112321 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.112382 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.112384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.113049 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.113050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.113577 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.113579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.113976 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.113977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.114061 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.114514 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.114526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.115784 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.117341 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.117342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.117343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.117514 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.117730 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.117730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.117731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.117796 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.118044 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.118045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.118091 (+ 46us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.118181 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.118285 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 21:10:26.118706 29301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:26.119050 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.118951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38914)
0808 21:10:26.118998 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.119025 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260808 21:10:26.119437 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.119356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38916)
0808 21:10:26.119397 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.119422 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260808 21:10:26.119541 29301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:36409: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
I20260808 21:10:26.139977 29381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } attempt: 1
I20260808 21:10:26.140558 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.140355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38918)
0808 21:10:26.140446 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.140501 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 21:10:26.140928 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.140835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38932)
0808 21:10:26.140881 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.140908 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260808 21:10:26.141022 29381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:36409: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
I20260808 21:10:26.150700 29301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } attempt: 1
I20260808 21:10:26.151103 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.150990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38942)
0808 21:10:26.151048 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.151081 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260808 21:10:26.151448 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.151369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38954)
0808 21:10:26.151410 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.151433 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260808 21:10:26.151551 29301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:36409: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
W20260808 21:10:26.180728 29388 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 21:10:26.180929 29388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 21:10:26.180946 29388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 21:10:26.180991 29388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 21:10:26.182118 29388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 21:10:26.182158 29388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 21:10:26.182173 29388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 21:10:26.182224 29388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 21:10:26.182245 29388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 21:10:26.183789 29388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.177.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.177.126:38123,127.4.177.125:32873,127.4.177.124:36409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/krb5kdc/kudu_127.4.177.124.keytab
--principal=kudu/127.4.177.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.177.124:36409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.177.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 08 Aug 2026 20:09:30 UTC on ccycloud.quasar-nuvkld.root.comops.site
build id 4575432
I20260808 21:10:26.184026 29388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 21:10:26.184290 29388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 21:10:26.189276 29396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:26.189347 29388 server_base.cc:1034] running on GCE node
W20260808 21:10:26.189298 29393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 21:10:26.189291 29394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 21:10:26.189731 29388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 21:10:26.190001 29388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 21:10:26.191180 29388 hybrid_clock.cc:648] HybridClock initialized: now 1786223426191170 us; error 31 us; skew 500 ppm
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 21:10:26.193574 29388 init.cc:377] Logged in from keytab as kudu/127.4.177.124@KRBTEST.COM (short username kudu)
I20260808 21:10:26.194017 29388 webserver.cc:460] Webserver started at http://127.4.177.124:41063/ using document root <none> and password file <none>
I20260808 21:10:26.194250 29388 fs_manager.cc:359] Metadata directory not provided
I20260808 21:10:26.194316 29388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 21:10:26.194478 29388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 21:10:26.195572 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data/instance:
uuid: "2c0ab76d68694de98b1d785888af8183"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "76c7422f82b9380504a3cc6d2d9685e3"
server_key_iv: "37b65856b53f105da88c0e4e8f917882"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.195940 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal/instance:
uuid: "2c0ab76d68694de98b1d785888af8183"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "76c7422f82b9380504a3cc6d2d9685e3"
server_key_iv: "37b65856b53f105da88c0e4e8f917882"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.197242 29388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260808 21:10:26.197947 29403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 21:10:26.198124 29388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260808 21:10:26.198200 29388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data,/tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal
uuid: "2c0ab76d68694de98b1d785888af8183"
format_stamp: "Formatted at 2026-08-08 21:10:26 on dist-test-slave-chsk"
server_key: "76c7422f82b9380504a3cc6d2d9685e3"
server_key_iv: "37b65856b53f105da88c0e4e8f917882"
server_key_version: "encryptionkey@0"
I20260808 21:10:26.198251 29388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 21:10:26.208750 29388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 21:10:26.209180 29381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } attempt: 2
I20260808 21:10:26.209573 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.209441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38964)
0808 21:10:26.209504 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.209542 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260808 21:10:26.210002 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.209892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38978)
0808 21:10:26.209939 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.209975 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260808 21:10:26.210146 29381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.177.124:36409: Network error: Client connection negotiation failed: client connection to 127.4.177.124:36409: connect: Connection refused (error 111)
I20260808 21:10:26.210958 29388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 21:10:26.211095 29388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 21:10:26.214990 29388 rpc_server.cc:307] RPC server started. Bound to: 127.4.177.124:36409
I20260808 21:10:26.215027 29459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.177.124:36409 every 8 connection(s)
I20260808 21:10:26.215485 29388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/data/info.pb
I20260808 21:10:26.216475 29460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 21:10:26.218019 29460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:26.223246 4805 external_mini_cluster.cc:1388] Started /tmp/dist-test-task496WfL/build/release/bin/kudu as pid 29388
I20260808 21:10:26.223373 4805 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task496WfL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786223155551886-4805-0/minicluster-data/master-2/wal/instance
I20260808 21:10:26.223574 4805 external_mini_cluster.cc:1428] Setting key 5ced6805a893122f2e89e64707bcafc9
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:26.226702 29301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } attempt: 2
I20260808 21:10:26.227465 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.218705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.4.177.126:38123)
0808 21:10:26.218801 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.218803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.218953 (+ 150us) server_negotiation.cc:407] Connection header received
0808 21:10:26.219509 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.219510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.219521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.219585 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.220184 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.220670 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.221661 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.221862 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.221948 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.224363 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.224364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.224365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.224373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.225914 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.226341 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.226342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.226343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.226394 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.226684 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.226685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.226686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.226818 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.226884 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.227272 (+ 388us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.227356 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 21:10:26.227669 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.218657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.126:38123 (local address 127.0.0.1:33092)
0808 21:10:26.218838 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.218873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.219403 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.219663 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.219666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.219908 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.220090 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.220093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.220766 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.220767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.221533 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.221535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.221970 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.221971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.222100 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.222622 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.222630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.224277 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.225994 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.226001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.226003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.226265 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.226460 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.226461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.226461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.226595 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.226905 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.226912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.227072 (+ 160us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.227249 (+ 177us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.227438 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260808 21:10:26.228638 29460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.126@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:26.234397 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.223986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.4.177.124:36409)
0808 21:10:26.224170 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.224174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.224195 (+ 21us) server_negotiation.cc:407] Connection header received
0808 21:10:26.224271 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.224274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.224287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.224365 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.225111 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.226169 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.227538 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.228060 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.228164 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.231198 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.231205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.231206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.231212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.233124 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.233528 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.233531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.233532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.233590 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.233781 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.233782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.233782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.233938 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.234112 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.234170 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.234279 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260808 21:10:26.236074 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.226941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:38996)
0808 21:10:26.227022 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.227031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.227066 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.227353 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.227354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.227522 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.227559 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.227561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.228479 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.228480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.229231 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.229234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.229787 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.229788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.229843 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.230226 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.230231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.231625 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.234888 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.234889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.234890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.235100 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.235363 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.235364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.235364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.235439 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.235709 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.235710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.235761 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.235839 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.235955 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260808 21:10:26.236709 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.227032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.4.177.124:36409)
0808 21:10:26.227217 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.227219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.227226 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:10:26.227245 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.227245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.227255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.227629 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.227669 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.228383 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.229334 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.229697 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.229764 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.231694 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.231695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.231696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.231702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.234820 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.235215 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.235216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.235217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.235283 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.235508 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.235509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.235509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.235616 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.235704 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.236460 (+ 756us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.236557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260808 21:10:26.236847 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.228967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.4.177.125:32873)
0808 21:10:26.229017 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.229019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.229027 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:10:26.229051 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.229052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.229061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.229109 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.229407 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.230303 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.231284 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.231574 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.231659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.233614 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.233616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.233616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.233625 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.235605 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.235966 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.235967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.235968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.236025 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.236277 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.236278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.236278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.236435 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.236503 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.236677 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.236755 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260808 21:10:26.236940 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.228877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.125:32873 (local address 127.0.0.1:47968)
0808 21:10:26.228931 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.228943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.228984 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.229135 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.229136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.229267 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.229343 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.229344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.230398 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.230400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.231205 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.231208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.231712 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.231713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.231792 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.232178 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.232183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.233514 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.235672 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.235673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.235674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.235889 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.236094 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.236095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.236095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.236172 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.236508 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.236509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.236547 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.236618 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.236696 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260808 21:10:26.238379 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: Bootstrap starting.
I20260808 21:10:26.239233 29301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:26.239223 29460 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } has no permanent_uuid. Determining permanent_uuid...
I20260808 21:10:26.239585 29301 log.cc:824] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: Log is configured to *not* fsync() on all Append() calls
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.126@KRBTEST.COM
I20260808 21:10:26.240451 29301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: No bootstrap required, opened a new log
I20260808 21:10:26.242262 29301 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.242404 29301 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ffb76082a14ad5b793e05a69fe0fca, State: Initialized, Role: FOLLOWER
I20260808 21:10:26.242497 29301 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.242800 29469 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.242877 29301 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:26.242893 29469 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.243577 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.235707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.4.177.126:38123)
0808 21:10:26.235748 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.235749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.235756 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:10:26.235773 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.235773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.235783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.235840 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.236447 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.237158 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.238031 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.238204 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.238258 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.240838 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.240839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.240840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.240847 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.242146 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.242492 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.242494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.242494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.242574 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.242780 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.242780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.242781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.242916 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.243094 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.243361 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.243457 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 21:10:26.244865 29480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:26.244920 29480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:26.245678 29231 master_runner.cc:186] Error getting master registration for 127.4.177.125:32873: OK, instance_id { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" instance_seqno: 1786223426066410 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:10:26.248247 29231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.124@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.177.125@KRBTEST.COM
I20260808 21:10:26.251734 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.239500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:39010)
0808 21:10:26.239891 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.239902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.239978 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.240086 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.240087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.244474 (+ 4387us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.244519 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.244522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.245413 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.245414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.246017 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.246019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.246528 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.246529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.246574 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.247112 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.247118 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.248862 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.250530 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.250531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.250532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.250755 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.250962 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.250963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.250963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.251021 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.251387 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.251387 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.251426 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.251495 (+ 69us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.251612 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260808 21:10:26.251735 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.239537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.4.177.124:36409)
0808 21:10:26.239568 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.239570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.239965 (+ 395us) server_negotiation.cc:407] Connection header received
0808 21:10:26.240033 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.240034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.240042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.240086 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.244619 (+ 4533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.245325 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.246103 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.246404 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.246478 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.248949 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.248950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.248950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.248957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.250432 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.250831 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.250832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.250833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.250883 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.251093 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.251094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.251094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.251315 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.251395 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.251520 (+ 125us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.251653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":17}
I20260808 21:10:26.252864 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.244239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.4.177.125:32873)
0808 21:10:26.244287 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.244288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.244296 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:10:26.244320 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.244320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.244330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.244383 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.245009 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.245685 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.246588 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.246906 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.246966 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.249717 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.249718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.249718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.249726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.251654 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.252059 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.252060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.252061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.252130 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.252335 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.252336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.252336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.252457 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.252638 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.252675 (+ 37us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.252752 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260808 21:10:26.253475 29460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: Bootstrap starting.
I20260808 21:10:26.254101 4805 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 21:10:26.254244 29460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:26.254573 29460 log.cc:824] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:26.254942 29275 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d6ffb76082a14ad5b793e05a69fe0fca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33098
I20260808 21:10:26.255119 29275 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 21:10:26.255215 29275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:26.255523 29460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: No bootstrap required, opened a new log
I20260808 21:10:26.256513 29275 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.256775 29275 leader_election.cc:288] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [CANDIDATE]: Term 1 election: Requested vote from peers d9f94941782a4972b67c04c3eaf03750 (127.4.177.125:32873), 2c0ab76d68694de98b1d785888af8183 (127.4.177.124:36409)
I20260808 21:10:26.257831 29460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.258010 29460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0ab76d68694de98b1d785888af8183, State: Initialized, Role: FOLLOWER
I20260808 21:10:26.258111 29460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.258414 29460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:26.258455 29433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c0ab76d68694de98b1d785888af8183"
I20260808 21:10:26.258587 29433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:26.259488 29433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ffb76082a14ad5b793e05a69fe0fca in term 1.
I20260808 21:10:26.259711 29248 leader_election.cc:302] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0ab76d68694de98b1d785888af8183, d6ffb76082a14ad5b793e05a69fe0fca; no voters:
I20260808 21:10:26.259831 29469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 1 FOLLOWER]: Leader election won for term 1
I20260808 21:10:26.259974 29469 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [term 1 LEADER]: Becoming Leader. State: Replica: d6ffb76082a14ad5b793e05a69fe0fca, State: Running, Role: LEADER
I20260808 21:10:26.260057 29469 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
W20260808 21:10:26.260481 29501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 21:10:26.261137 29501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 21:10:26.261809 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ffb76082a14ad5b793e05a69fe0fca. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.262138 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:26.262346 29388 master_runner.cc:186] Error getting master registration for 127.4.177.125:32873: OK, instance_id { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" instance_seqno: 1786223426066410 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 21:10:26.264590 29388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 21:10:26.264744 29506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 21:10:26.264940 29506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 21:10:26.266501 29433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [term 1 FOLLOWER]: Refusing update from remote peer d6ffb76082a14ad5b793e05a69fe0fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:10:26.266798 29503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:10:26.268445 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ffb76082a14ad5b793e05a69fe0fca. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.268714 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.269160 29508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.269372 29508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.270656 29469 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.270738 29469 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:26.271312 29507 mvcc.cc:205] Tried to move back new op lower bound from 7316371153984786432 to 7316371153961848832. Current Snapshot: MvccSnapshot[applied={T|T < 7316371153984786432 or (T in {7316371153984786432})}]
I20260808 21:10:26.271420 29506 catalog_manager.cc:1347] Generated new cluster ID: ac82cd91bdf64176bb63c36005e357d5
I20260808 21:10:26.271451 29506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 21:10:26.275214 29508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.275317 29508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.278925 29511 mvcc.cc:205] Tried to move back new op lower bound from 7316371153984786432 to 7316371153961848832. Current Snapshot: MvccSnapshot[applied={T|T < 7316371153984786432}]
I20260808 21:10:26.285679 29506 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 21:10:26.288301 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.268199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.4.177.125:32873)
0808 21:10:26.268243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.268245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.271285 (+ 3040us) server_negotiation.cc:407] Connection header received
0808 21:10:26.271391 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.271392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.271403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.271435 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.275654 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.276420 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.277549 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.278444 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.278513 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.282814 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.282815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.282816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.282823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.284869 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.286751 (+ 1882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.286753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.286753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.286813 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.287329 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.287329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.287330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.287469 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.287561 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.288125 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.288207 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:10:26.288959 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.267652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.4.177.124:36409)
0808 21:10:26.267702 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.267704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.279283 (+ 11579us) server_negotiation.cc:407] Connection header received
0808 21:10:26.279370 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.279371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.279380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.279408 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.280106 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.280818 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.281771 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.282103 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.282186 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.283578 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.283579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.283579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.283586 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.287427 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.287840 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.287841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.287842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.287908 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.288111 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.288112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.288112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.288240 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.288333 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.288782 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.288872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260808 21:10:26.289628 29506 catalog_manager.cc:1504] Loading token signing keys...
I20260808 21:10:26.290709 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.258075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.4.177.126:38123)
0808 21:10:26.258127 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.258129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.268427 (+ 10298us) server_negotiation.cc:407] Connection header received
0808 21:10:26.268583 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.268585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.268608 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.268657 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.275712 (+ 7055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.276472 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.281047 (+ 4575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.281333 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.281372 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.284631 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.284633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.284633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.284639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.286660 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.289608 (+ 2948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.289609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.289609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.289787 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.290049 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.290050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.290051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.290172 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.290260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.290521 (+ 261us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.290610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:10:26.301278 29381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } attempt: 3
Aug 08 21:10:26 dist-test-slave-chsk krb5kdc[29218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786223426, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.177.125@KRBTEST.COM for kudu/127.4.177.124@KRBTEST.COM
I20260808 21:10:26.312294 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.301555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.177.124:36409 (local address 127.0.0.1:39036)
0808 21:10:26.301660 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0808 21:10:26.301672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0808 21:10:26.301711 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 21:10:26.302067 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 21:10:26.302068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 21:10:26.302259 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0808 21:10:26.302308 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.302310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.303337 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.303338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.304174 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 21:10:26.304177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.305487 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.305488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 21:10:26.305549 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.306136 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 21:10:26.306142 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 21:10:26.307701 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 21:10:26.310785 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.310786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.310787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.311042 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.311346 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 21:10:26.311347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 21:10:26.311347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0808 21:10:26.311417 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 21:10:26.311983 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 21:10:26.311984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 21:10:26.312040 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 21:10:26.312089 (+ 49us) client_negotiation.cc:240] Negotiation successful
0808 21:10:26.312189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 21:10:26.312837 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:26.301783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.4.177.124:36409)
0808 21:10:26.301876 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.301878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.301885 (+ 7us) server_negotiation.cc:407] Connection header received
0808 21:10:26.301922 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.301923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.301932 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.301979 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.302418 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.303225 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.304316 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:26.304591 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:26.304662 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:26.307805 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 21:10:26.307807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 21:10:26.307807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 21:10:26.307814 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0808 21:10:26.310655 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.311125 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.311126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.311126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.311265 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 21:10:26.311497 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 21:10:26.311497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 21:10:26.311498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 21:10:26.311837 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 21:10:26.312510 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:26.312575 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 21:10:26.312701 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 21:10:26.314508 29381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: Bootstrap starting.
I20260808 21:10:26.315505 29381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: Neither blocks nor log segments found. Creating new log.
I20260808 21:10:26.315832 29381 log.cc:824] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: Log is configured to *not* fsync() on all Append() calls
I20260808 21:10:26.316816 29381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: No bootstrap required, opened a new log
I20260808 21:10:26.318933 29381 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.319077 29381 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f94941782a4972b67c04c3eaf03750, State: Initialized, Role: FOLLOWER
I20260808 21:10:26.319190 29381 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } }
I20260808 21:10:26.319520 29381 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 21:10:26.319700 29353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [term 0 FOLLOWER]: Advancing to term 1
I20260808 21:10:26.320621 29353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [term 1 FOLLOWER]: Refusing update from remote peer d6ffb76082a14ad5b793e05a69fe0fca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 21:10:26.320897 29354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d9f94941782a4972b67c04c3eaf03750"
I20260808 21:10:26.321686 29354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ffb76082a14ad5b793e05a69fe0fca in term 1.
I20260808 21:10:26.322140 29503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 21:10:26.325683 29506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca: Generated new TSK 0
I20260808 21:10:26.325591 29531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.325793 29531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.325861 29531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ffb76082a14ad5b793e05a69fe0fca. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.325918 29531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.325958 29531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.326018 29531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 21:10:26.326115 29506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 21:10:26.327232 29534 catalog_manager.cc:1259] Loaded cluster ID: ac82cd91bdf64176bb63c36005e357d5
I20260808 21:10:26.327199 29469 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ffb76082a14ad5b793e05a69fe0fca" member_type: VOTER last_known_addr { host: "127.4.177.126" port: 38123 } } peers { permanent_uuid: "d9f94941782a4972b67c04c3eaf03750" member_type: VOTER last_known_addr { host: "127.4.177.125" port: 32873 } } peers { permanent_uuid: "2c0ab76d68694de98b1d785888af8183" member_type: VOTER last_known_addr { host: "127.4.177.124" port: 36409 } } }
I20260808 21:10:26.327275 29534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: loading cluster ID for follower catalog manager: success
I20260808 21:10:26.327289 29469 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ffb76082a14ad5b793e05a69fe0fca [sys.catalog]: This master's current role is: LEADER
I20260808 21:10:26.329293 29534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: acquiring CA information for follower catalog manager: success
W20260808 21:10:26.329473 29534 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260808 21:10:26.363034 29304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.361155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.4.177.126:38123)
0808 21:10:26.361204 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.361206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.362281 (+ 1075us) server_negotiation.cc:407] Connection header received
0808 21:10:26.362398 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.362400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.362412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.362445 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.362955 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33118: BlockingRecv error: recv got EOF from 127.0.0.1:33118 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260808 21:10:26.364046 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.361613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.4.177.125:32873)
0808 21:10:26.361659 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.361661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.363501 (+ 1840us) server_negotiation.cc:407] Connection header received
0808 21:10:26.363614 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.363615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.363626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.363657 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.363985 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47994: BlockingRecv error: recv got EOF from 127.0.0.1:47994 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260808 21:10:26.364955 29466 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.361946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.4.177.124:36409)
0808 21:10:26.361987 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.361988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.364458 (+ 2470us) server_negotiation.cc:407] Connection header received
0808 21:10:26.364552 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.364554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.364565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.364595 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.364900 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39050: BlockingRecv error: recv got EOF from 127.0.0.1:39050 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260808 21:10:26.368487 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.367708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48004 (local address 127.4.177.125:32873)
0808 21:10:26.367746 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.367748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.367964 (+ 216us) server_negotiation.cc:407] Connection header received
0808 21:10:26.368075 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.368075 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.368094 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.368125 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.368441 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48004: BlockingRecv error: recv got EOF from 127.0.0.1:48004 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 21:10:26.371744 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.370932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.4.177.125:32873)
0808 21:10:26.370966 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.370968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.371238 (+ 270us) server_negotiation.cc:407] Connection header received
0808 21:10:26.371336 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.371337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.371347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.371377 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.371702 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48016: BlockingRecv error: recv got EOF from 127.0.0.1:48016 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260808 21:10:26.375753 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.374948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.4.177.125:32873)
0808 21:10:26.374982 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.374983 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.375239 (+ 256us) server_negotiation.cc:407] Connection header received
0808 21:10:26.375336 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.375337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.375347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.375377 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.375711 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48030: BlockingRecv error: recv got EOF from 127.0.0.1:48030 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260808 21:10:26.381937 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.381109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.4.177.125:32873)
0808 21:10:26.381148 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.381150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.381392 (+ 242us) server_negotiation.cc:407] Connection header received
0808 21:10:26.381492 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.381493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.381504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.381533 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.381889 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48044: BlockingRecv error: recv got EOF from 127.0.0.1:48044 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 21:10:26.392376 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.391462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.4.177.125:32873)
0808 21:10:26.391502 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.391504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.391780 (+ 276us) server_negotiation.cc:407] Connection header received
0808 21:10:26.391885 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.391886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.391899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.391931 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.392304 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48050: BlockingRecv error: recv got EOF from 127.0.0.1:48050 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260808 21:10:26.410924 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.409938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.4.177.125:32873)
0808 21:10:26.409984 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.409986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.410279 (+ 293us) server_negotiation.cc:407] Connection header received
0808 21:10:26.410390 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.410392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.410405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.410436 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.410864 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48058: BlockingRecv error: recv got EOF from 127.0.0.1:48058 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260808 21:10:26.445420 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.444644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.4.177.125:32873)
0808 21:10:26.444706 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.444708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.444860 (+ 152us) server_negotiation.cc:407] Connection header received
0808 21:10:26.444943 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.444944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.444956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.445004 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.445362 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48072: BlockingRecv error: recv got EOF from 127.0.0.1:48072 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260808 21:10:26.511606 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.510834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.4.177.125:32873)
0808 21:10:26.510938 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.510941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.511060 (+ 119us) server_negotiation.cc:407] Connection header received
0808 21:10:26.511130 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.511130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.511165 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.511213 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.511547 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48084: BlockingRecv error: recv got EOF from 127.0.0.1:48084 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 21:10:26.641809 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.640955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.4.177.125:32873)
0808 21:10:26.641035 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.641037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.641133 (+ 96us) server_negotiation.cc:407] Connection header received
0808 21:10:26.641253 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.641257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.641272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.641362 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.641742 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48088: BlockingRecv error: recv got EOF from 127.0.0.1:48088 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260808 21:10:26.899873 29385 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 21:10:26.899157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.4.177.125:32873)
0808 21:10:26.899237 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:26.899239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:26.899249 (+ 10us) server_negotiation.cc:407] Connection header received
0808 21:10:26.899291 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:26.899292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:26.899301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:26.899378 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 21:10:26.899804 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48098: BlockingRecv error: recv got EOF from 127.0.0.1:48098 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260808 21:10:27.261571 29501 catalog_manager.cc:1259] Loaded cluster ID: ac82cd91bdf64176bb63c36005e357d5
I20260808 21:10:27.261659 29501 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: loading cluster ID for follower catalog manager: success
I20260808 21:10:27.262480 29501 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: acquiring CA information for follower catalog manager: success
I20260808 21:10:27.262814 29501 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c0ab76d68694de98b1d785888af8183: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:10:27.330053 29534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d9f94941782a4972b67c04c3eaf03750: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 21:10:27.415694 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:27.413159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.4.177.125:32873)
0808 21:10:27.413275 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:27.413277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:27.413288 (+ 11us) server_negotiation.cc:407] Connection header received
0808 21:10:27.413338 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:27.413339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:27.413398 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:27.413475 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:27.413621 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.414115 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.414883 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.415076 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.415132 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:27.415342 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:27.415448 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:27.415510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:27.415548 (+ 38us) server_negotiation.cc:299] Negotiation successful
0808 21:10:27.415567 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260808 21:10:27.420060 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:27.417672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.4.177.124:36409)
0808 21:10:27.417797 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:27.417799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:27.417807 (+ 8us) server_negotiation.cc:407] Connection header received
0808 21:10:27.417843 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:27.417844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:27.417877 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:27.417948 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:27.418033 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.418520 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.419283 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.419509 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.419563 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:27.419718 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:27.419817 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:27.419868 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:27.419938 (+ 70us) server_negotiation.cc:299] Negotiation successful
0808 21:10:27.419955 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260808 21:10:27.424496 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:27.421967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.4.177.124:36409)
0808 21:10:27.422005 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:27.422007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:27.422195 (+ 188us) server_negotiation.cc:407] Connection header received
0808 21:10:27.422221 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:27.422222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:27.422231 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:27.422283 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:27.422423 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.422882 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.423813 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.423999 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.424072 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:27.424228 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:27.424260 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:27.424321 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:27.424375 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 21:10:27.424399 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260808 21:10:27.424667 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:27.421817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.4.177.125:32873)
0808 21:10:27.421863 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:27.421866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:27.421875 (+ 9us) server_negotiation.cc:407] Connection header received
0808 21:10:27.421976 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:27.421977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:27.421988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:27.422064 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:27.422176 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.422639 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.423748 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.424032 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.424073 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:27.424309 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:27.424344 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:27.424396 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:27.424565 (+ 169us) server_negotiation.cc:299] Negotiation successful
0808 21:10:27.424593 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260808 21:10:27.425810 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 21:10:27.421825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.4.177.126:38123)
0808 21:10:27.421927 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0808 21:10:27.421929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0808 21:10:27.422111 (+ 182us) server_negotiation.cc:407] Connection header received
0808 21:10:27.422790 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 21:10:27.422791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 21:10:27.422841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 21:10:27.422924 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 21:10:27.423095 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.423869 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.425052 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 21:10:27.425358 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 21:10:27.425431 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 21:10:27.425568 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 21:10:27.425647 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 21:10:27.425692 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0808 21:10:27.425735 (+ 43us) server_negotiation.cc:299] Negotiation successful
0808 21:10:27.425754 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260808 21:10:27.431897 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 29231
I20260808 21:10:27.439532 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 29308
I20260808 21:10:27.445688 4805 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task496WfL/build/release/bin/kudu with pid 29388
2026-08-08T21:10:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2876 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5720 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271890 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST